Lecture 06

Reading material: Section 3.1 - 3.3 of CSSB.

Recap & Intro

Last time, we discussed correlations, the concept of orthogonality, inner products, the issue with correlations and the use of averaging to filter out noise.

In this lecture and the next we will try to tie together a few concepts introduced so far to understand the Fourier series and expansions.

Some background

We tend to think of a vector as a tuple or collection of real numbers. Intuitively, this makes sense to us because in two and three dimensions these correspond to the familiar coordinates from 2D and 3D geometry. Vector addition, multiplication etc. are thus intuitively familiar to us and we have no qualms with writing: let $x \in \mathbb{R}^3$ refer to a vector in 3D. In fact, we are even comfortable with: let $y \in \mathbb{R}^n$. Now let us take a step back and think of $\mathbb{R}^n$.

What is it?

Vector spaces

We naturally think of $\mathbb{R}^n$ as the space in which $x$ or $y$ or our vectors live in or belong to - a vector space. How is this formalized? Well, recall from BIOE210^[1] that we have field axioms that specify the rules followed by elements that we can do algebra with. Similarly, there are axioms or rules that a space $V$ must satisfy to be a vector space. Note that there is no specification in the field axioms on what the elements $a, b$ and $c$ are: only that they follow some rules.

In the same vein, there are no restrictions on what we can consider as vectors in a vector space as long as they follow these rules. Don't be intimidated by the math on that page: you already know a few vector spaces! For example $\mathbb{R}, \mathbb{R}^2$ and $\mathbb{R}^n$ are all vector spaces which follow those rules; only that the rules are written up in an abstract way.

Basis

Now follows a slightly wordy definition:

Take note of how we have written a summation to express $v$ in terms of the basis. This way of writing vectors will show up a lot.

Whenever we talked of vectors, we have been implicitly using a basis – in fact the standard basis $\{e_k\}$ which are defined as vectors that are zero in all entries except the $k$-th one. Thus even a vector like $v = [3, 4, 5]^T$ is actually $v = 3e_1 + 4e_2 + 5e_3$ where

As mentioned above, the choice of the basis vector set $B$ is not unique. It just so happens that the set $\{e_k\}_{k=1}^n$ as the basis for $\mathbb{R}^n$ is extremely convenient. The construction below shows how the same point in $\mathbb{R}^2$ can be represented in two different bases.

The black axes obviously represent the standard basis $e_1$ and $e_2$. The blue pair of rays represent the directions given by a new set of basis vectors $B$ (in black) that you can choose by entering the vectors $a$ and $b$ into the input boxes. Since we are in 2D or equivalently $\mathbb{R}^2$, we need a pair of basis vectors $B=\{a, b\}$ and each vector has two entries, $a=[a_x, a_y]^T$ and $b=[b_x, b_y]^T$.

Then moving the blue point around shows how it is represented in coordinates that utilize the usual basis (in black) and the newly chosen basis (in blue). Thus the blue coordinates & the black coordinates represent the same vector in $\mathbb{R}^2$ but with different sets as the basis.

This non-uniqueness of the basis set is not a peculiar feature of just $\mathbb{R}^2$ or $\mathbb{R}^n$.

As shown above $P_n(x)$ has the familiar basis $\{1, x, x^2, x^3, \dots, x^n\}$ that we are used to seeing. But we can also find another basis set. For example consider the polynomials in $x$ given by

One can show that the set $\{L_k(x)\}_{k=1}^n$ is also a basis for $P_n(x)$.

Why do we care?

Apart from providing a preview of what will be a major & crucial topic in any course on linear algebra^[2]; the main takeaway here is that representing a problem or situation in a different basis can often make things simpler or provide more insight into a problem. For example polar & spherical coordinates often simplify problems in physics as you may recall.

Indeed, many important techniques you will learn, including Principal Components Analysis, Singular Value Decomposition, eigendecompositions etc. are all essentially a change of basis to some particularly useful basis vectors. Moreover, change of basis calculations form an extremely important part of various analytical & problem solving techniques including solving differential equations, implementing matrix multiplications, etc.

More pertinently for us, recall that we can think of signals and functions as being elements of certain vector space. It turns out that the in the continuous domain, what is akin to the change-of-basis we discussed above are called integral transforms $\dots$ but more on that later.

Fourier Transform(s)

CSSB talks about four different "transforms" in the forward and reverse directions^[3]. They are listed as:

Please take care to use the right name because each of the techniques have slightly different areas of applicability as listed in the table. In the words of one bestselling author:

You might be thinking that the names given to these four types of Fourier transforms are confusing and poorly organized. You're right; the names have evolved rather haphazardly over 200 years. There is nothing you can do but memorize them and move on.
~ S. W. Smith

We will start with the DFT above, and discuss each in turn, though we may not follow the unwieldy notation used in CSSB. We start with the DFT because it is directly related to the discussions we have had above.

Discrete Fourier Transform (DFT)

Given one period of discrete periodic signal $f[n]$ as vector of length $N$ we define the DFT of it as the sequence of $N$ complex numbers given by:

Note the similarity of the above summation with those we have been using to express vectors in terms of different basis. This is no coincidence because the inverse DFT is:

We finally hit pay dirt for mining through all the math so far.

One naturally wonders why we would bother to write a signal in terms of complex exponentials. Here is where we tie up another thread we introduced a couple of lectures back via Euler's formula: complex exponentials are sinusoids!

Thus in the above:

One should think of this as evaluating a sine and cosine of frequency $\omega_k$ at the time $n$.

Note that the summation is over $k$ in (3): Therefore, we are expressing the original signals as a summation of sinusoids of different frequencies! This is useful because sinusoids are the "nicest" periodic functions in a sense^[4].

Fourier's great insight was that theoretically, any periodic function can be expressed in terms of a summation of sines & cosines $+$ a constant. In practice though, there are a few caveats. For example, the function to be expressed needs to be "nice" in certain mathematical terms. Sometimes the summation needed might be an infinite one which we cannot do on a computer. The demonstration below illustrates this with a square wave. The first collection of plots (of different colors) show the sinusoids of five different frequencies being added together to create the plot immediately below it (green). The green plot can be changed by using the checkboxes to add/remove sine wave of a particular frequency. The last plot (brown) shows what happens when you increase the number of frequencies being summed together (using the slider). At its maximum value, we see that we get a very good approximation of a square wave.

In the above definition we took a complete period of a signal $f[k]$ when we computed its transform. This is an implicit assumption whenever we perform the DFT. In practice, it is rarely the case that we will have one complete period of the signal we are performing DFT on which leads to some artifacts $\dots$. These will be the subject of subsequent lectures.

Note that while we discuss other flavors of the transform next; the only one we can implement and perform on real world signals is the DFT via a computer. Nevertheless, the other transforms are important to know for purposes of analysis and problem solving and we look at these next.

Fourier Series

In the above discussion, we had a discrete periodic signal to begin with. What happens if instead we have a continuous periodic signal? We no longer have any indices to sum over; so we should adjust our equations a bit. While in terms of implementation, everything in a computer relies on the DFT, conceptually we tend to think in terms of continuous signals and so this is a practical concern.

To go from discrete to continuous recall that we summed over a complete period of a signal. For continuous signals the summation becomes an integration and just like before we integrate over a full period $T$. Next, the summation variable $n$ turns into the integration variable $t$. Moreover as we replace $f[t] \mapsto f(t)$ we also do:

Thus we get that Fourier Series corresponding to a periodic signal $f(t)$ over one period $[0, T]$ is given as:

where we have normalized by the period $T$. The quantity $\omega_0$ should be familiar to us from our discussion of periodic signals and is in the context of Fourier analysis called the fundamental frequency. Moreover, in literature it is common to call (4) and (2) as forward or analysis equations and the inverse transform as in (3) as the synthesis equation because these synthesize our signal out of sinusoids. Thus, for a continuous periodic signal, the synthesis equation is given by:

Note that the synthesis equation is still a summation and the analysis equations generates an infinite sequence of Fourier coefficients $F[k]$. This explains the reason why we call it the Fourier Series. Ideally, we should have called the DFT the Discrete Fourier Series but ... oh well 🤷.

Obviously, only a fraction of the signals we see in real life are periodic signals. In our discussion so far, we have assumed the signals to be periodic regardless of whether discrete or continuous. Next, we will try to relax this assumption.

Fourier Transform

As stated previously in Lecture 02 one convenient way to think of aperiodic signals is to treat them as signals whose period $T \to \infty$. To extend the equations above to signals that are not periodic, this is exactly what we do. We let $T$ go to $\infty$ and let the machinery for mathematical theory of limits take over. We don't work this out here but one can easily see that as $T \to \infty$ we have $\omega_0 \to 0$. Yet, in the integration, we have $k \omega_0$ with $k= 0, \pm 1, \pm 2, \dots$ all the way to $\infty$. Thus the term $k \omega_0$ does not go to zero, rather becomes a continuous variable $\omega$ and we perform the necessary integration over the whole real line. Then we have for the forward equation:

In tandem, since $k \omega _0$ became a continuous variable $\omega$, it is now necessary to replace the sum in (5) with an integral. Therefore we get the synthesis equation:

Intuitively, one can think of it like this: if a periodic signal (like the square wave) requires an infinite summation of sines and cosines of discrete frequencies to represent it, then an aperiodic signal would require even more – an infinite summation of continuous frequencies; in fact an integral.

Wrap up

At this point, it might be instructive to take a minute and recall that we remarked in Lecture 01 that signals admit equivalent time and frequency domain representations and referred to this figure below. It is precisely via the Fourier transform that we get the frequency domain representation. Each coordinate on the right panel gives us three things: a frequency, a phase and a magnitude corresponding to a single sinusoid in the (possibly infinite) summation required to represent it.

We will skip the DTFT because of its limited practical application and in the next lecture we will arrive at equations equivalent to the ones shown in this lecture but without using imaginary numbers and complex mathematics.

In this lecture note, the intention was to provide a birds eye view of the theory of Fourier transforms and how it fits in the greater scheme of things. We have grossly neglected to discuss any implementations or calculations; both of which, hopefully will be addressed by homework, and in particular the latter which we will rectify with some worked out examples.

[back]