Teaching Staff
- M.-A. Belabbas, Office: CSL 166, email:
belabbas@illinois.edu
Assistant: Stephanie McCullough, CSL 153.
Office hours: Tuesday 5-6.
Lectures and recordings
The class meets Tuesday and Thursday 3:30pm to 4:50pm at Zoom Link
You can find the recording of the lectures here.
Notes
Notes will be posted here
Course notes (they may be updated)
Chap 1-2 (Review or Probability and introduction to Stochastic Processes, Half-page format)
Chap 3 (Poisson counters and Ito differential equations, Half-page format)
Chap 4 (Optimal control, Markov decision processes, Half-page format)
Chap 5 (Wiener process and Ito differential equations, Half-page format)
Chap 6 (System Concepts: Fourier Transform, Wiener-Khinchin, Stochastic realization)
Chap 7 (Nonlinear filtering)
Chap 8 (LQG control and separation principle)
Chap 9 (Birkhoff Ergodic Theorem and abstract theory of Markov processes)
Lecture 1: Introduction and review of probability theory. (Chap 1. Random variables, sigma fields, probability spaces, measurability and adapted sigma fields) Notes
Lecture 2: Introduction to stochastic processes (Chap 2. Filtration, adapted process, the simple random walk, Gambler’s ruin, Stationarity.)
Lecture 3: Conditioning of a random variable with respect to a sigma-field. Martingales. Stopping times.
Lecture 4: Martingales. Stopping times. Optional Stopping Theorem. Wald’s inequality. Doob’s and Ville’s martingale inequalities.
Lecture 5: Borrowing is good. Poisson counters, their expectation and distribution. Exponential distribution of jump times. Stochastic differential equation driven by Poisson processes.
Lecture 6: Stochastic differential equation driven by Poisson processes. Ito rule. Expectation rule. Computing the moments of a solution of an SDE.
Lecture 7: Finite state-space, continuous time Markov Chains, their probabilistic and sample paths description. Hitting times for Markov chains. Embedded chain.
Lecture 8: Expected hitting times for finite, continuous time Markov Chains. Fokker-Planck Equation. Fluid queues. Computing time correlations.
Lecture 9: Optimality principle, Hamilton-Jacobi-Bellman equation for deterministic, discrete-time processes.
Lecture 10: Markov Decision Processes. Optimal stopping time and secretary problem.
Lecture 11: Infinite horizon dynamic programming for Markov decision process. Discounted, negative and positive programs. Value iteration.
Lecture 12: Dynamic programming in continuous time for MDP and deterministic systems. LQ regulator.
Lecture 13: Wiener process from Poisson counters. Brownian motion and heat equation. Moments of a Wiener process.
Lecture 14: Stochastic differential equations driven by a Wiener process. Ito rule. Expectation rule. Ornstein-Uhlenbeck process. Numerical integration via Euler-Marayuma method.
Lecture 15: Milstein method for numerical integration. Differentiability of Wiener process. Iterated integrals.
Lecture 16: Fokker-Planck equation for diffusion-driver stochastic differential equation (Forward Kolmogorov equation). Stratonovic Calculus.
Lecture 17: Review of Linear Systems Theory. Fourier Transform, Power Spectrum. Ergodic processes.
Lecture 18: Power Spectrum. Wiener-Khinchin Theorem. Stochastic realization.
Lecture 19: Filtering and Smoothing of time series.
Lecture 20: Conditional density equation (Duncan-Mortsensen-Zakai equation).
Lecture 21: Kalman-Bucy filter. Conditional expectation as optimal least square estimator.
Lecture 22: Kalman-Bucy filter as optimal unbiased linear filter. Optimal LQG control. Separation principle.
Lecture 23: Maximal inequality for measure preserving ergodic processes. (If watching videos, Lec 24 is recommended before Lec 23)
Lecture 24: Measure preserving dynamics. Poincaré recurrence theorem. Ergodic dynamics.
Lecture 25: Ergodic dynamics, Koopman operators.
Lecture 26: Ergodic measures, ergodic decomposition theorem.
Lecture 27: Kac’s lemma. von Neumann ergodic theorem
Lecture 28: Stochastic processes and Kolmogorov’s extension theorem
Homeworks
There will be roughly 5-6 homeworks assigned.
Exams
- There are no exams.
Project
The final product is a report (about 5-10 pages) on an application requiring some of the methods presented in the class or a summary of a paper/group of papers related to the material covered in class. A summary/overview of something not covered in class (e.g., Malliavin calculus) is also acceptable.
The due date is May 16 2022.
Matlab
We will use Matlab at various points in this class, but the use will never be too involved. It is assumed that you are at least mildly proficient with the program.
Textbook
There are no mandatory textbooks. I will distribute notes as we go along.
Prerequisites
The official prerequisites for this course are ECE 515 (Control system theory and design) and ECE 534 (Random Processes). In practice, a good understanding of linear algebra, probability and the willingness to do additional work on the way to cover possible gaps in your background should be enough. If you fall in this latter category, please discuss first with me.
Homeworks
Homeworks will be posted here