University of Illinois Urbana-Champaign · Fall 2026

ECE 490Introduction to Optimization

A proof-oriented introduction to continuous optimization for seniors/first year graduate students. Mathematical foundations of unconstrained and constrained optimization, convexity vs nonconvexity, analysis of large-scale modern algorithms.

Instructor
Richard Y. Zhang — ryz@illinois.edu
Teaching Assistant
Hong-Ming Chiu — hmchiu2@illinois.edu
Class
ECEB 3081, Tue and Thu, 11:00 AM–12:20 PM
Office Hours
CSL 115, Thu, 12:30–1:30 PM, or by appointment
Short URL
ryz.nz/490

Course

Course introduction

Tue Aug 25
Motivation and Formulation: Why optimization? Why take a proof-based class? Finding vs certifying. Basic problem classes; decision variables, objectives, and constraints. Formulation vs solution. Worked examples: formulate least-squares problems from verbal descriptions.

Foundational Overview

Thu Aug 27
Algorithmic Overview: How to find a good solution? Implement GD, SGD, momentum, and Newton's method. For constraints, quadratic penalties and log barriers. Worked examples: optimizer updates, one Newton step, penalty and barrier formulations.BlankFilled
Tue Sep 1
Convexity and PSD Matrices: How to certify a good solution? Convexity by definition and by the Hessian test. Basic combination rules. Worked examples: prove and disprove convexity using both criteria.BlankFilled
Thu Sep 3
Convex Sets and Projected Gradient: What about constraints? Convex sets by line segments and sublevel sets. Intersections and affine transformations. Projected gradient descent. Worked examples: prove and disprove set convexity; perform one projected-gradient step.BlankFilled
Tue Sep 8Quiz 1: Foundational overviewHomeworkSolutions

Gradient Descent and First-order Optimality

Thu Sep 10
Optimality Conditions: Where can gradient descent converge? First-order necessary and convex sufficient conditions. Infimum versus minimum; boundedness below, continuity, and coercivity. Second-order necessary and sufficient conditions, and benign nonconvexity. Worked examples: solve least squares by first-order optimality; test coercivity; classify first- and second-order critical points.BlankFilled
Tue Sep 15
Fixed-Step Gradient Descent: How fast does gradient descent converge? L-smoothness and the Hessian test. The descent lemma and the step size 1/L. Sublinear convergence to first-order optimality; linear convergence under the PL condition; objective convergence for strongly convex functions. Worked examples: compute smoothness constants and valid step sizes; derive convergence rates by telescoping the descent inequality.BlankFilled
Thu Sep 17
Line Search and Convergence: How should we choose the step size? Best fixed and changing steps, exact line search, and Armijo backtracking. Finite termination and a uniform lower bound on accepted steps. Convergence to first-order optimality, and faster objective convergence under PL and strong convexity. Worked examples: compute exact line search for a quadratic; carry out backtracking by hand; repeat the fixed-step convergence proofs with sufficient decrease.BlankFilled
Tue Sep 22Quiz 2HomeworkSolutions

Newton's Method and Second-order Optimality

Thu Sep 24
Newton’s Method: Why Newton's method? Linear versus quadratic convergence and first- versus second-order critical points. Newton as inverse-Hessian preconditioned gradient descent; exact solution of strictly convex quadratics in one step. Local quadratic convergence under strong convexity and Lipschitz Hessian, and why full-step Newton can fail away from the solution.BlankFilled
Tue Sep 29
Globalizing Newton: How can Newton converge from a poor initial point? Backtracking Newton and the Newton decrement. Taylor bounds from Lipschitz continuity of the Hessian. Fixed-decrement and quadratic-convergence phases, giving global convergence for strongly convex problems. Trust-region methods as an alternative way to control unreliable quadratic models.BlankFilled
Thu Oct 1
Second-order optimality: How can we find second-order critical points in nonconvex problems? Why first-order optimality is insufficient and second-order optimality is useful on benign landscapes. Cubic-regularized Newton from a global upper bound on the quadratic Taylor model. One-step decrease, gradient, and curvature bounds, leading to convergence to approximate second-order critical points.BlankFilled
Tue Oct 6Quiz 3: Newton's method and second-order optimalityHomework

Midterm Exam

Thu Oct 8Midterm Review
Tue Oct 13Midterm Exam

Duality and Infeasibility Certification

Thu Oct 15
Tue Oct 20
Thu Oct 22
Tue Oct 27Quiz 4: Duality and infeasibility certification

Local Optimality under Constraints

Thu Oct 29
Tue Nov 3
Thu Nov 5
Tue Nov 10Quiz 5: Local optimality under constraints

Constrained and Nonsmooth Algorithms

Thu Nov 12
Tue Nov 17
Thu Nov 19
Tue Nov 24Fall Break — No Class
Thu Nov 26Fall Break — No Class
Tue Dec 1Quiz 6: Constrained and nonsmooth algorithms

Final Exam

Thu Dec 3Cumulative Review
Tue Dec 8Reading Week — No Class
Thu Dec 10Reading Week — No Class
Fri Dec 11Final exam — 8am to 11am — ECE 3081

Grading, homework, quizzes, and exams

  • 0% - Biweekly homework
  • 50% - Biweekly quizzes (top 5 out of 6 scores)
  • 25% - Midterm exam
  • 25% - Final exam

Homework is ungraded and serves as the problem bank for quizzes and exams. Quiz and exam problems are sampled directly from the homework.

Many homework problems are marked (G). These problems are treated differently in the 3-credit and 4-credit sections, as described below.

Quizzes

Each quiz contains 4 problems sampled from the current homework, including 1 problem marked (G). Quizzes last 50 minutes and are held in class approximately biweekly on Tuesdays. The first 30 minutes of class before each quiz are used for review.

For 3-credit students, the three non-(G) problems make up the regular score and the (G) problem is optional bonus. Partial credit on the (G) problem can replace points lost on the regular problems, but the quiz score is capped at 100%.

For 4-credit students, the (G) problem is required and the lowest-scoring non-(G) problem is dropped.

Example quiz

Problem 3-credit section 4-credit section
P1 Required Eligible for drop
P2 Required Eligible for drop
P3 Required Eligible for drop
P4 (G) Optional bonus Required

Let P1, P2, P3, P4 denote the normalized score from 0 to 1.

  • 3-credit score: min( (P1 + P2 + P3 + P4)*100/3, 100 )
  • 4-credit score: ( P4 + sum of the two highest scores among P1, P2, P3 )*100/3

The lowest quiz score is dropped to accommodate travel and other occasional absences. It is in your interest to attend all quizzes because of the make-up policy below.

Midterm exam

The midterm contains 6 problems sampled from the first three homeworks, including 2 problems marked (G). It lasts 80 minutes and is held in class.

For 3-credit students, the exam is graded out of 5 problems. Both (G) problems may be attempted, and all earned credit counts toward the score before it is capped at 100%. In this sense, one (G) problem is optional, but partial credit on it can still improve the score.

For 4-credit students, both (G) problems are required and the lowest-scoring non-(G) problem is dropped.

Let P1, P2, P3, P4 denote the normalized scores on the four non-(G) problems, and G1, G2 the normalized scores on the two (G) problems.

  • 3-credit score: min( (P1 + P2 + P3 + P4 + G1 + G2)*100/5, 100 )
  • 4-credit score: ( G1 + G2 + sum of the three highest scores among P1, P2, P3, P4 )*100/5

Final exam

The final contains 8 problems sampled from the homework: 6 problems from the last three course blocks and 2 problems from the first three course blocks. Two of the 8 problems are marked (G). The final lasts 180 minutes and is scheduled in the regular classroom.

For 3-credit students, the exam is graded out of 7 problems. Both (G) problems may be attempted, and all earned credit counts toward the score before it is capped at 100%. In this sense, one (G) problem is optional, but partial credit on it can still improve the score.

For 4-credit students, both (G) problems are required and the lowest-scoring non-(G) problem is dropped.

Let P1, P2, P3, P4, P5, P6 denote the normalized scores on the six non-(G) problems, and G1, G2 the normalized scores on the two (G) problems.

  • 3-credit score: min( (P1 + P2 + P3 + P4 + P5 + P6 + G1 + G2)*100/7, 100 )
  • 4-credit score: ( G1 + G2 + sum of the five highest scores among P1, P2, P3, P4, P5, P6 )*100/7

Missed assessments

Exceptional circumstances may necessitate make-up quiz or exam. The reasoning must be consistent with the student code.

Makeup quiz requests will only be considered after a student has been absent for one quiz. If a student has not been absent for a quiz, then the makeup request will be automatically declined.

The student must contact course staff with timestamp no later than 7 days (168 hours, 0 minutes, 0 seconds) before the starting time of the quiz or exam to inquire about make-up assessments.

AI policy

AI tools are strongly encouraged for studying and for working through the ungraded homework. Useful ways to use AI include:

  • asking for a different explanation of a theorem or proof;
  • requesting a hint when stuck;
  • asking AI to identify a gap in an argument;
  • checking algebra or intermediate calculations;
  • asking why a proposed proof is invalid;
  • generating additional problems equivalent to a homework problem;
  • asking for a complete solution and then working backward until every step can be reproduced independently.

Illinois students have access to Google Gemini through their University Google accounts. Illinois Google Gemini information

AI systems can make mathematical mistakes. Students remain responsible for checking AI output.

AI may not be used during any quiz or examination.

University policies

Disability-related accommodations

The University provides reasonable academic accommodations to students with disabilities through Disability Resources and Educational Services (DRES). Students seeking accommodations should register with DRES and provide their Letter of Academic Accommodations to the instructor as early as possible.

Approved accommodations will be implemented in accordance with Student Code § 1-110. Student Code § 1-110 — Disability accommodations

Religious observances

The University will reasonably accommodate religious beliefs, observances, and practices that conflict with class attendance, quizzes, examinations, or other course requirements.

Students should request accommodation sufficiently far in advance to allow alternate arrangements to be made. See Student Code § 1-107. Student Code § 1-107 — Religious accommodations

Emergency response

Students should familiarize themselves with University emergency procedures and with the evacuation and shelter procedures for ECEB. Instructions from University and emergency personnel should be followed during an emergency.

Other University policies

All applicable University of Illinois policies remain in effect, including policies concerning student privacy, nondiscrimination, community standards, sexual misconduct, student well-being, and military or veteran obligations. Students should consult the current University of Illinois Student Code for the governing policies and procedures.