0:00

So we were talking about instances where the design matrix X was orthonormal,

Â it had orthonormal columns, so that X transpose x was the identity matrix.

Â And let's talk about some of the most famous cases where this happens.

Â Easily the biggest one is the so called Fourier basis.

Â So in the Fourier basis, the columns of X are trigonometric.

Â So imagine if your y is a time series.

Â And then the columns of x are trigonometric terms,

Â basically sines and cosines of different periods,

Â from slow, low varying signals

Â to really high-frequency signals toward the end of the x matrix.

Â 0:50

So what this means is that the coefficients

Â associated from a least squares fit, from these basis elements and

Â my observed time series y,

Â the coefficients are just the inner product of these specific columns.

Â Let's say xi is column i from x, and y.

Â And so the transformation that takes y to

Â the collection of these basis elements, in other words,

Â that takes y to x transpose y, that transformation

Â is called the Fourier transformation, or the discrete Fourier transformation.

Â In fact, it can be done faster then this calculation.

Â It can be done capitalizing on some of the redundancies

Â in trigonometric relationships.

Â It can be done faster to result in the so-called fast discrete Fourier transform.

Â What you get with the Fourier transform is,

Â each coefficient is the amount of sort of the signal

Â that's correlated with a sine or a cosine turn of that frequency.

Â So as an example, if I had a signal that looked like this.

Â 2:10

So it looks like maybe a sine term and a cosine term of,

Â a slow variation sine term and a high-frequency term added together,

Â then those two terms, the coefficients associated with those two terms,

Â will wind up having big coefficients when we do x transpose y.

Â 2:35

So when we want to reconstruct the signal again, that's just summation,

Â right, the basis elements times the inner product of the x's and the y's.

Â So we can do tricks, for example, if we want to reconstruct the signal,

Â but only omit, only include those terms that are very low frequency.

Â So for example, if we want this big rolling term but we want to get rid of

Â the high frequency information, we might do what is called a low pass filter.

Â So we would let the low-frequency terms through, and

Â we would filter out the high-frequency terms.

Â So this would amount to, just in our reconstructed signal here,

Â only adding those components associated with the low-frequency terms in the basis.

Â And conversely, we might want to just capture the high-frequency stuff and

Â filter out the low-frequency stuff.

Â So in this sum, we would just take the high-frequency terms and omit the others.

Â And then the reconstructed signal would have filtered out the low-frequency

Â information.

Â So again, this all boils down to least squares, like we've been studying.

Â However, I think it's fair to say that signal processors tend to

Â think of this in a different way.

Â But it's nice to put this very important concept, Fourier analysis,

Â in the domain of least squares, which we've been talking about in this class.

Â The second basis that I would mention to discuss a little bit is so-called

Â wavelet bases.

Â And wavelet bases are similar to Fourier bases.

Â They're an orthonormal basis that have some nice properties associated with them.

Â And there is a discrete wavelet transform,

Â just like there's a discrete Fourier transform.

Â But it's actually able to get the transform faster than if you were to do

Â it by calculating each of these inner products by themselves.

Â 4:34

Again, the Fourier basis and the wavelet basis, though, to really discuss them

Â we'd have to get into the nitty-gritty of the construction of the bases.

Â So I think I'm going to only then cover, really, in a little bit more detail, only

Â one of the three cases that's a little bit more statistical, one of the three most

Â important cases of orthonormal bases that's a little bit more statistical.

Â And that's so-called principal component bases.

Â So in the next lecture, we're going to talk about a particularly important

Â version of a basis where x transpose x = I.

Â