ECE 563  Information Theory (Fall 2020)
Lecturer: Lav Varshney (office hours, Friday 9:30am10:30am, Zoom)
Teaching Assistant: Sourya Basu (office hours, Wednesday, 9:00am10:00am, Zoom)
Lectures: Tuesday and Thursday, 12:30pm, Zoom (if you have not received the password, please ask the course staff). Recordings via Illinois Media Space.
Problem Solving Sessions: Monday, 9:00am10:00am, Zoom [optional]
Course Goals
Catalog Description
Mathematical models for channels and sources; entropy, information, data compression, channel capacity, Shannon's theorems, and ratedistortion theory.
Prerequisites: Solid background in probability (ECE 534, MATH 464, or MATH 564).
Textbook: T. M. Cover and J. A. Thomas, Elements of Information Theory, 2nd ed., Wiley, 2006.
Grading:
Homework (all via GradeScope, if you have not received invitation, ask course staff)
Problem Solving Sessions
Old exams
Exams
Juxtaposition Paper
Course Schedule
Date  Topic  Reading Assignment  Learning Objectives  Multimedia Supplements 
8/25 
1. The problem of communication, information theory beyond communication [slides] 



8/27 
2. The idea of errorcontrol coding and linear codes 


9/1 
3. Information measures and their axiomatic derivation 



9/3 
4. Basic inequalities with information measures 



9/8 
5. Asymptotic Equipartition Property 



9/10 
6. Source Coding Theorem 



9/15 
7. Variablelength Codes 


9/17 
8. Entropy Rate of Stochastic Processes 


9/22 
9. Distributed Source Coding 


9/24  10. Universal Source Coding 



9/29  11. Method of Types 


10/1  12. Exam 1 [no lecture]  
10/6  13. Hypothesis Testing 


10/8  14. Channel Coding Theorem: Converse and Joint AEP 



10/13  15. Channel Coding Theorem: Achievability and Examples 


10/15  16. SourceChannel Separation 


10/20  17. Differential Entropy, Maximum Entropy, and Capacity of RealValued Channels 


10/22  18. RateDistortion Theorem: Converse and Examples 



10/27  19. Exam 2 [no lecture] 



10/29  20. RateDistortion Theorem: Achievability and More Examples 


11/3  21. Election Day [no lecture] 



11/5  22. Quantization Theory 


11/10  23. BlahutArimoto 


11/12  24. Strong Data Processing Inequalities 


11/17  25. Large Deviations 


11/19  26. Error Exponents for Channel Coding 


12/1  27. Error Exponents for Channel Coding 


12/3  28. Multiple Access Channel: Achievability 


12/8  29. Multiple Access Channel: Converse, Examples, and Duality 
