ECE 563 - Information Theory (Fall 2025)

Lecturer: Olgica Milenkovic (Office hours: Thursday 2:30-3:30pm, 311 CSL or by appointment as needed)

Teaching Assistants: Peizhi Niu (Office hour: Tuesday 3:00-4:00pm, Electrical and Computer Engineering Building ECEB 3036; peizhin2@illinois.edu)

Lectures: Tuesday and Thursday, 12:30-13:50, Electrical and Computer Engineering Building ECEB 3081

Course Objectives:

• To learn about various forms of entropy and generalizations thereof, and their operational interpretations/characterizations.
• To understand how one can characterize the fundamental performance limits of diverse communication systems and learning models.
• To explore connections between information theory, statistics and machine learning.
• To learn about emerging applications of information theory in compression, learning, computing, etc.

Catalog Description

Mathematical models for channels and sources; entropy, information, data compression, channel capacity, Shannon's theorems, rate-distortion theory. Information theory and statistics, learning.

Prerequisites: Solid background in probability (ECE 534, MATH 464, or MATH 564).

Required textbook: T. M. Cover and J. A. Thomas, Elements of Information Theory, 2nd ed., Wiley, 2006.

Recommended textbook: Y. Poliyanskiy and Y. Wu, Information Theory: From Coding to Learning, Cambridge University Press, 2025.

Grading: Homework will be assigned and solutions provided, but not graded, Midterm I exam: October 7th, 6:30-8:00 pm, Electrical and Computer Engineering Building ECEB 2015 (25%) CLOSED NOTES, Midterm II exam: November 6th (Tentative) (25%) CLOSED NOTES,  Final exam: Dec. XXX, 2024 (50%).

Homework Submission: Please submit your homework to the designated Box folder corresponding to the assignment: Box Link. Each submission should be named using the format NetID_HW# (for example: peizhin2_HW1). Please upload only a single PDF file per assignment. Make sure to upload your file before the deadline.

Syllabus

Homework, Fall 2025

• Homework 1 Issued September 9th, due September 18th in class. Solutions. The reference paper by Renyi for Problem 1 can be found here.
• Homework 2 Issued September 22nd, due October 2nd in class. Solutions.
• Homework 3 To be issued October 7th, due October 16th in class. 
• Homework 4 To be issued October  21st, due October 30th in class. 
• Homework 5 To be issued November 4th, due November 13th in class. 
• Homework 6 To be issued November 18th, due November 27th in class. 

Additional Instructional Material

Entropy in Physics (Video, TEDed)  

Operational Characterization of Entropy (Video, Khan Academy)  

Lecture subjects, Fall 2025

1. Tuesday, August 26th: Introduction, Syllabus Overview, How to measure information in physics, engineering, communication theory.

The part of the first lecture pertaining to entropy in physics and how it inspired Shannon's entropy can be found at Notes 1. The axiomatic derivation of Shannon's entropy from class is based on R. Ash, Information Theory, pp. 5-12. More on axiomatic approaches can be found here Entropy axioms  

2. Thursday, August 28th: Axioms of Shannon entropy, derivation of the Shannon entropy function through an axiomatic approach, generalized means, Renyi entropy

A summary of the notes pertaining to generalized means and Renyi entropy can be found at Notes 2.