ECE 563 - Information Theory (Fall 2019)
Lecturer: Olgica Milenkovic (Office hours: Thursday 3:30-5:00pm, 313 CSL or by appointment as needed)
Teaching Assistants: Pattabiraman, Srilakshmi (Office hours, Tuesday 3:00-4:00pm, 3036 ECE; sp16@illinois.edu)
Lectures: Tuesday and Thursday, 12:30pm, 2015 Electrical and Computer Engineering Building
Problem Solving Sessions: The sessions will start on September 13th, 2019 Friday, 2:00pm, 141 Coordinated Science Laboratory [optional]
Course Objectives:
Catalog Description
Mathematical models for channels and sources; entropy, information, data compression, channel capacity, Shannon's theorems, and rate-distortion theory.
Prerequisites: Solid background in probability (ECE 534, MATH 464, or MATH 564).
Textbook: T. M. Cover and J. A. Thomas, Elements of Information Theory, 2nd ed., Wiley, 2006.
Grading: Homework (25%), Midterm exam [in class] (25%), Final exam [at a date determined by the university] (25%), Group project/paper (25%)
Midterm I: October 24th, 6pm, Room 2017
Midterm I Review: October 22nd, 5:00-6:30pm, Room 2017
Project Presentations: December 6th, 12:00 noon - 5:00pm, Room 311 CSL
Research Project Topics
Genomic Data Compression Review paper
The Information Bottleneck Problem Research paper
Lovasz number of a graph Lecture notes
Quantized Deep Learning Blog
Capacity of DNA-Storage Channels Research paper
Polar codes Research paper
Network coding Text
Quantum information theory Text
Renyi entropy NeurIPS paper
Deletion error-correction and capacity of deletion channels Review paper by Sloane
Channel dispersion: finite block-length regime Y. Polyansky et al., see also the prior work by Strassen.
Additional Instructional Material
Entropy in Physics (Video, TEDed)
Operational Characterization of Entropy (Video, Khan Academy)
The first lecture on the axiomatic derivation of Shannon's entropy is based on R. Ash, Information Theory, pp. 5-12. More on axiomatic approaches can be found here Entropy axioms
Homeworks, Fall 2019
Problem Solving Sessions, Fall 2019
Homeworks, Fall 2018 with Solutions
Problem Solving Sessions, Fall 2018
Exams
Juxtaposition Paper
Course Schedule
Date | Topic | Reading Assignment | Learning Objectives | Multimedia Supplements |
8/28 |
1. The problem of communication, information theory beyond communication [slides] |
|
||
8/30 |
2. The idea of error-control coding and linear codes [slides] [handwritten] |
|
||
9/4 | 3. Information measures and their axiomatic derivation |
|
|
|
4. Basic inequalities with information measures |
|
|||
9/11 | 5. Asymptotic Equipartition Property |
|
|
|
9/13 | 6. Source Coding Theorem |
|
|
|
9/18 | 7. Variable-length Codes |
|
||
9/20 | 8. Entropy Rate of Stochastic Processes |
|
||
9/25 | 9. Distributed Source Coding |
|
||
9/27 | 10. Universal Source Coding |
|
|
|
10/2 | 11. Method of Types |
|
||
10/4 | 12. Allerton Conference [no lecture] | |||
10/9 | 13. Hypothesis Testing |
|
||
10/11 | 14. Channel Coding Theorem: Converse and Joint AEP |
|
|
|
10/16 | 15. Channel Coding Theorem: Achievability and Examples |
|
||
10/18 | 16. Midterm [no lecture] | |||
10/23 | 17. Source-Channel Separation |
|
||
10/25 | 18. Differential Entropy, Maximum Entropy, and Capacity of Real-Valued Channels |
|
||
10/30 | 19. Rate-Distortion Theorem: Converse and Examples |
|
|
|
11/1 | 20. Rate-Distortion Theorem: Achievability and More Examples |
|
||
11/6 | 21. Quantization Theory |
|
||
11/8 | 22. Blahut-Arimoto |
|
||
11/13 | 23. Strong Data Processing Inequalities |
|
||
11/15 | 24. Large Deviations |
|
||
11/27 | 25. Error Exponents for Channel Coding |
|
||
11/29 | 26. Error Exponents for Channel Coding |
|
||
12/4 | 27. Multiple Access Channel: Achievability |
|
||
12/6 | 28. Quantum Information Theory [guest lecture] | |||
12/11 | 29. Multiple Access Channel: Converse, Examples, and Duality |
|
Topics: