ECE 490Introduction to Optimization
A proof-oriented introduction to continuous optimization for seniors/first year graduate students. Mathematical foundations of unconstrained and constrained optimization, convexity vs nonconvexity, analysis of large-scale modern algorithms.
Course
Course introduction
| Tue Aug 25 | Motivation and Formulation: Why optimization? Why take a proof-based class? Finding vs certifying. Basic problem classes; decision variables, objectives, and constraints. Formulation vs solution. Worked examples: formulate least-squares problems from verbal descriptions. |
Foundational Overview
| Thu Aug 27 | Algorithmic Overview: How to find a good solution? Implement GD, SGD, momentum, and Newton's method. For constraints, quadratic penalties and log barriers. Worked examples: optimizer updates, one Newton step, penalty and barrier formulations. |
| Tue Sep 1 | Convexity and PSD Matrices: How to certify a good solution? Convexity by definition and by the Hessian test. Basic combination rules. Worked examples: prove and disprove convexity using both criteria. |
| Thu Sep 3 | Convex Sets and Projected Gradient: What about constraints? Convex sets by line segments and sublevel sets. Intersections and affine transformations. Projected gradient descent. Worked examples: prove and disprove set convexity; perform one projected-gradient step. |
| Tue Sep 8 | Quiz 1: Foundational overview |
Gradient Descent and First-order Optimality
| Thu Sep 10 | Optimality Conditions: Where can gradient descent converge? First-order necessary and convex sufficient conditions. Infimum versus minimum; boundedness below, continuity, and coercivity. Second-order necessary and sufficient conditions, and benign nonconvexity. Worked examples: solve least squares by first-order optimality; test coercivity; classify first- and second-order critical points. |
| Tue Sep 15 | Fixed-Step Gradient Descent: How fast does gradient descent converge? L-smoothness and the Hessian test. The descent lemma and the step size 1/L. Sublinear convergence to first-order optimality; linear convergence under the PL condition; objective convergence for strongly convex functions. Worked examples: compute smoothness constants and valid step sizes; derive convergence rates by telescoping the descent inequality. |
| Thu Sep 17 | Line Search and Convergence: How should we choose the step size? Best fixed and changing steps, exact line search, and Armijo backtracking. Finite termination and a uniform lower bound on accepted steps. Convergence to first-order optimality, and faster objective convergence under PL and strong convexity. Worked examples: compute exact line search for a quadratic; carry out backtracking by hand; repeat the fixed-step convergence proofs with sufficient decrease. |
| Tue Sep 22 | Quiz 2 |
Newton's Method and Second-order Optimality
| Thu Sep 24 | Newton’s Method: Why Newton's method? Linear versus quadratic convergence and first- versus second-order critical points. Newton as inverse-Hessian preconditioned gradient descent; exact solution of strictly convex quadratics in one step. Local quadratic convergence under strong convexity and Lipschitz Hessian, and why full-step Newton can fail away from the solution. |
| Tue Sep 29 | Globalizing Newton: How can Newton converge from a poor initial point? Backtracking Newton and the Newton decrement. Taylor bounds from Lipschitz continuity of the Hessian. Fixed-decrement and quadratic-convergence phases, giving global convergence for strongly convex problems. Trust-region methods as an alternative way to control unreliable quadratic models. |
| Thu Oct 1 | Second-order optimality: How can we find second-order critical points in nonconvex problems? Why first-order optimality is insufficient and second-order optimality is useful on benign landscapes. Cubic-regularized Newton from a global upper bound on the quadratic Taylor model. One-step decrease, gradient, and curvature bounds, leading to convergence to approximate second-order critical points. |
| Tue Oct 6 | Quiz 3: Newton's method and second-order optimality |
Midterm Exam
| Thu Oct 8 | Midterm Review |
| Tue Oct 13 | Midterm Exam |
Duality and Infeasibility Certification
| Thu Oct 15 | |
| Tue Oct 20 | |
| Thu Oct 22 | |
| Tue Oct 27 | Quiz 4: Duality and infeasibility certification |
Local Optimality under Constraints
| Thu Oct 29 | |
| Tue Nov 3 | |
| Thu Nov 5 | |
| Tue Nov 10 | Quiz 5: Local optimality under constraints |
Constrained and Nonsmooth Algorithms
| Thu Nov 12 | |
| Tue Nov 17 | |
| Thu Nov 19 | |
| Tue Nov 24 | Fall Break — No Class |
| Thu Nov 26 | Fall Break — No Class |
| Tue Dec 1 | Quiz 6: Constrained and nonsmooth algorithms |
Final Exam
| Thu Dec 3 | Cumulative Review |
| Tue Dec 8 | Reading Week — No Class |
| Thu Dec 10 | Reading Week — No Class |
| Fri Dec 11 | Final exam — 8am to 11am — ECE 3081 |
Applets
Find vs Certify
Finding something is easy. Certifying it does not exist is hard.
1D Global Optim
When can you find the global optimum, and when can you certify it?
AI training as Nonconvex Opt
What do modern optimizers do in the face of nonconvexity?
Penalty vs Barrier
What is the best way to enforce a constraint? Soft penalty or hard barrier?
Saddle Points
Build saddle-point examples and study their derivatives and gradient-descent fields.
Convergence Rates
What is the difference between polynomial and exponential convergence? Sublinear, linear, and superlinear convergence?
Newton Globalization
When does Newton converge spectacularly fast, and when does it need safeguards?
Textbooks and references
Official textbook (not required for class)
Primary references (free access to PDF):
Grading, homework, quizzes, and exams
- 0% - Biweekly homework
- 50% - Biweekly quizzes (top 5 out of 6 scores)
- 25% - Midterm exam
- 25% - Final exam
Homework is ungraded and serves as the problem bank for quizzes and exams. Quiz and exam problems are sampled directly from the homework.
Many homework problems are marked (G). These problems are treated differently in the 3-credit and 4-credit sections, as described below.
Quizzes
Each quiz contains 4 problems sampled from the current homework, including 1 problem marked (G). Quizzes last 50 minutes and are held in class approximately biweekly on Tuesdays. The first 30 minutes of class before each quiz are used for review.
For 3-credit students, the three non-(G) problems make up the regular score and the (G) problem is optional bonus. Partial credit on the (G) problem can replace points lost on the regular problems, but the quiz score is capped at 100%.
For 4-credit students, the (G) problem is required and the lowest-scoring non-(G) problem is dropped.
Example quiz
| Problem | 3-credit section | 4-credit section |
|---|---|---|
| P1 | Required | Eligible for drop |
| P2 | Required | Eligible for drop |
| P3 | Required | Eligible for drop |
| P4 (G) | Optional bonus | Required |
Let P1, P2, P3, P4 denote the normalized score from 0 to 1.
- 3-credit score: min( (P1 + P2 + P3 + P4)*100/3, 100 )
- 4-credit score: ( P4 + sum of the two highest scores among P1, P2, P3 )*100/3
The lowest quiz score is dropped to accommodate travel and other occasional absences. It is in your interest to attend all quizzes because of the make-up policy below.
Midterm exam
The midterm contains 6 problems sampled from the first three homeworks, including 2 problems marked (G). It lasts 80 minutes and is held in class.
For 3-credit students, the exam is graded out of 5 problems. Both (G) problems may be attempted, and all earned credit counts toward the score before it is capped at 100%. In this sense, one (G) problem is optional, but partial credit on it can still improve the score.
For 4-credit students, both (G) problems are required and the lowest-scoring non-(G) problem is dropped.
Let P1, P2, P3, P4 denote the normalized scores on the four non-(G) problems, and G1, G2 the normalized scores on the two (G) problems.
- 3-credit score: min( (P1 + P2 + P3 + P4 + G1 + G2)*100/5, 100 )
- 4-credit score: ( G1 + G2 + sum of the three highest scores among P1, P2, P3, P4 )*100/5
Final exam
The final contains 8 problems sampled from the homework: 6 problems from the last three course blocks and 2 problems from the first three course blocks. Two of the 8 problems are marked (G). The final lasts 180 minutes and is scheduled in the regular classroom.
For 3-credit students, the exam is graded out of 7 problems. Both (G) problems may be attempted, and all earned credit counts toward the score before it is capped at 100%. In this sense, one (G) problem is optional, but partial credit on it can still improve the score.
For 4-credit students, both (G) problems are required and the lowest-scoring non-(G) problem is dropped.
Let P1, P2, P3, P4, P5, P6 denote the normalized scores on the six non-(G) problems, and G1, G2 the normalized scores on the two (G) problems.
- 3-credit score: min( (P1 + P2 + P3 + P4 + P5 + P6 + G1 + G2)*100/7, 100 )
- 4-credit score: ( G1 + G2 + sum of the five highest scores among P1, P2, P3, P4, P5, P6 )*100/7
Missed assessments
Exceptional circumstances may necessitate make-up quiz or exam. The reasoning must be consistent with the student code.
Makeup quiz requests will only be considered after a student has been absent for one quiz. If a student has not been absent for a quiz, then the makeup request will be automatically declined.
The student must contact course staff with timestamp no later than 7 days (168 hours, 0 minutes, 0 seconds) before the starting time of the quiz or exam to inquire about make-up assessments.
AI policy
AI tools are strongly encouraged for studying and for working through the ungraded homework. Useful ways to use AI include:
- asking for a different explanation of a theorem or proof;
- requesting a hint when stuck;
- asking AI to identify a gap in an argument;
- checking algebra or intermediate calculations;
- asking why a proposed proof is invalid;
- generating additional problems equivalent to a homework problem;
- asking for a complete solution and then working backward until every step can be reproduced independently.
Illinois students have access to Google Gemini through their University Google accounts. Illinois Google Gemini information
AI systems can make mathematical mistakes. Students remain responsible for checking AI output.
AI may not be used during any quiz or examination.
University policies
Disability-related accommodations
The University provides reasonable academic accommodations to students with disabilities through Disability Resources and Educational Services (DRES). Students seeking accommodations should register with DRES and provide their Letter of Academic Accommodations to the instructor as early as possible.
Approved accommodations will be implemented in accordance with Student Code § 1-110. Student Code § 1-110 — Disability accommodations
Religious observances
The University will reasonably accommodate religious beliefs, observances, and practices that conflict with class attendance, quizzes, examinations, or other course requirements.
Students should request accommodation sufficiently far in advance to allow alternate arrangements to be made. See Student Code § 1-107. Student Code § 1-107 — Religious accommodations
Emergency response
Students should familiarize themselves with University emergency procedures and with the evacuation and shelter procedures for ECEB. Instructions from University and emergency personnel should be followed during an emergency.
Other University policies
All applicable University of Illinois policies remain in effect, including policies concerning student privacy, nondiscrimination, community standards, sexual misconduct, student well-being, and military or veteran obligations. Students should consult the current University of Illinois Student Code for the governing policies and procedures.