|
ECE 598IS - Information Theory and High-Dimensional Statistics (Fall 2025)
Department of Electrical and Computer Engineering
University of Illinois Urbana-Champaign
Lectures: Monday and Wednesday, 2:00PM - 3:20PM, location: 2015 ECEB
Instructor: Ilan Shomorony (he/him), ilans@illinois.edu, 313 CSL
Announcements
- Homework 1 has been posted here
- The scribe sign-up sheet can be found here
- Scribe Latex template
Summary:
This course will expose students to a variety of theoretical tools that are useful in the context of high-dimensional probability and statistics. Students will learn to analyze the behavior of high-dimensional random vectors and matrices, and to utilize these insights to study statistical tasks motivated by large-scale datasets. We will explore how techniques from information theory can be used to derive converse results, and how techniques from random matrix theory and graph theory can be used to design estimators with theoretical guarantees. Topics will include information measures and inequalities, concentration inequalities, information-theoretic lower bounding techniques, sparse signal recovery, large-scale regression via leverage scores, inference tasks on large graphs, and dimensionality reduction.
Prerequisite: Main prerequisite is ECE 534 (Random Processes) or equivalent. ECE 563 (Information Theory) or some prior exposure to information theory thinking is recommended.
Textbook: No textbook will be required. Lectures will be based on the following sources:
- Roman Vershynin, "High-Dimensional Probability"
- Martin Wainwright, "High-Dimensional Statistics: A Non-Asymptotic Viewpoint"
- Yihong Wu, "Information-theoretic Methods for High-dimensional Statistics" (lecture notes)
- John Duchi, "Information Theory and Statistics" (lecture notes)
- Joel Tropp, "An Introduction to Matrix Concentration Inequalities"
- Alessandro Rinaldo, "Advanced Statistical Theory I" (lecture notes)
- Rodrigues and Eldar, "Information-Theoretic Methods in Data Science"
Tentative outline (subject to changes):
1. Statistics and Information Theory
- Statistical decision theory
- Bayes and minimax risk
- Information measures, divergences, and inequalities
- Le Cam's two-point method and Fano's method
- Sparse recovery and compressed sensing
2. Random matrices and graphs
- Matrix concentration inequalities
- Eigenvalue and eigenvector perturbations
- The planted clique problem, Stochastic Block Model
- Spectral estimator, SDP-based estimator
3. Dimensionality reduction
- Johnson-Lindenstrauss, Random projections
- Large-scale linear regression and linear algebra via randomization
- PCA on estimated covariance matrix, Sparse PCA
- Sketching and Locality-Sensitive Hashing
Grading
- Scribing - 15%
- Homework - 50%
- Project - 35%
|