0:23

We will decompose sounds into these two parts.

Â The sinusoidal or harmonic one and the residual or

Â ideally a stochastical one if this residual is the stochastic signal.

Â So in this lecture,

Â we will be combining all the models we have been talking about until now.

Â We'll put together the sinusoidal or

Â harmonic modelling with the idea of residual component.

Â For that we will need to talk about the subtraction of the sinusoidal,

Â the harmonics in order to obtain the residual.

Â And we will talk about a system that puts these together into the harmonic

Â plus residual system.

Â Then we will introduce the stochastic model and

Â we'll put together the sinusoidal and or

Â harmonic models with the stochastic ones for the residual.

Â So in order to do that, we'll need to talk about how to model

Â the residual as in a stochastic component.

Â And finally we will make an example of this system that combines

Â the harmonic plus the stochastic analysis into an analysis synthesis system.

Â 3:28

So let's show an example exactly how these will work.

Â This is a one frame of a sound and here we can show the different

Â steps involved these harmonic plus a residual analysis.

Â On the left top, we see our window frame of a flute sound, okay?

Â So it's just few periods of a flute sound.

Â And then below that, we see the harmonic analysis that we do from the spectrum.

Â So we do the spectral analysis the peaks, and we select the peaks with these

Â blue crosses that are really the harmonics of that particular sound.

Â And below it we see the actual phase of the spectrum with the crosses and

Â the phase of these harmonics, okay?

Â And then what we do is we have to synthesize these harmonics.

Â [COUGH] And this is what we see on the right side with the light

Â 4:36

red and light cyan color.

Â So the light red is the synthesized harmonics of

Â the sound of that particular frame.

Â Of course this is a different FFT size,

Â the shapes of these lobes is different because the window is different.

Â This is a window using the synthesis.

Â So this is the synthesized spectrum and

Â then we have to subtract these spectrum from the original spectrum.

Â Strictly speaking, we don't subtract it from the spectrum on the left,

Â we subtract this from another generated spectrum that has the same parameters so

Â that we can subtract the two of the same size and the same window size.

Â And then it will subtract this synthesize sinusoids or

Â harmonics from the original one.

Â We get this dark red and dark cyan color.

Â Okay, and this is the residual spectrum in magnitude and phase representation.

Â And if we take the inverse of that, we see the residual signal in the tandem and

Â that's what we see on the top-right plot.

Â In which we see the original flute sound of course with

Â the right windowing and the right size that we have in the synthesis.

Â And we see the residual signal, this dark blue one.

Â And again, this is not just an error signal,

Â this in fact is a relevant component,

Â it's a relevant part of the sound that we want to recover.

Â So the whole system, if we put together all this analysis in a frame by

Â frame type of thing and put it together into a whole analysis synthesis system.

Â We get this block diagram in which we start from the signal x[n],

Â then we window it, we compute FFT, obtaining the magnitude and

Â phase spectrum, we detect the peaks.

Â Hold up those peaks, we find the fundamental frequency, and

Â once we have this fundamental frequency we can identify the harmonics of that sound.

Â And we can synthesize those harmonics with the window.

Â Okay, so we have another spectrum,

Â Yh, that can be subtracted from the original signal.

Â But in order to do that,

Â we need to recompute another spectrum of the original with a window and

Â a size that is comparable with the size that we use in the synthesis.

Â So we will choose a window size that normally will be 512 samples,

Â we'll use a window so that the shape of these X[k]

Â that we now compute can be easily subtracted From Yh.

Â So it's just a complex subtraction and we get capital Xr,

Â which is our residual spectrum, okay?

Â And then this residual spectrum can be added to the harmonic spectrum.

Â And then, we can compute the inverse FFT and

Â do the Overlap-add iterating over the whole sum.

Â We can see an example of the analysis of a particular sound using the harmonic

Â class residual model.

Â So here, we took the flute sound that we have heard before, and

Â so on top, we see the spectrogram of these flute sound and

Â superpose we see the harmonics that have been obtained.

Â So let's listen to these harmonic synthesis.

Â [SOUND] Okay, and then these harmonics are subtracted

Â from this background spectrogram and we obtain these

Â lower sort of a plot which are the spectrogram of the residual component.

Â So let's just now listen to this residual that has been obtained.

Â [NOISE] Okay, it's soft but it's clearly very relevant.

Â It's basically the breath noise of the instrument which is

Â an important part of the characteristics of the sound.

Â But this residual component is a complete sound.

Â 9:45

So very similar to what we saw before.

Â So we have the signal to be the sum of

Â sinusoids plus the stochastic signal.

Â Now this is stochastic signal or stochastic component is not just

Â the subtraction of the sinusoids minus the original signal but

Â it's actually the result of a modeling approach.

Â So, below here, we see the equation of the modeling of this stochastic component.

Â The stochastic component is the result of filtering wide noise with the impulse

Â response of the approximation of these residual signal.

Â So we have the impulse response of every frame of this residual signal,

Â and we obtain this impulse response that approximates a spectral shape of that.

Â So in fact, it's much better to visualize this model in the frequency domain.

Â And so here we see on the top, the equation of the sum of the sinusoid,

Â the sum of the analysis windows plus the spectrum of the stochastic component.

Â And now this stochastic component is this idea of

Â a filter white noise but in the frequency domain,

Â is the product of the approximation of the absolute

Â value of the residual signal multiplied by e to the j and

Â the phase of the random, the white noise set of random numbers, okay?

Â So, the magnitude spectrum is the approximation of the residual and

Â the phase spectrum is the white noise.

Â Basically, the phase spectrum of the white noise, this is the concept of

Â the stochastic approximation that we saw in the previous lecture, okay?

Â So, with these, we can actually see how in a single spectrum,

Â we actually perform this stochastic approximation of the residual.

Â So we start on the top with the spectrum of a signal, the harmonics.

Â And then below the light red is

Â the synthesized spectrum, the mYh.

Â And then this is subtracted from the original spectrum,

Â again it's a spectrum that will have to be recomputed.

Â And then we obtained the next curve which is

Â the mXr which is the residual spectrum, okay?

Â And this residual spectrum can be approximated with a smooth

Â curve which is the mYst, which is this sort of

Â line approximation of this residual.

Â And this is going to be our stochastic model.

Â 12:54

So, we can put it together into an analysis synthesis system and

Â it's very similar to what we saw before.

Â So we start from the signal.

Â We compute FFT, we find the peaks, we find the harmonics,

Â we synthesize them in the frequency domain and we subtract them from

Â another spectrum of the original signal recomputed to be able to subtract it.

Â And then what is new in this model is the stochastic approximation of the residual.

Â So we take this residual spectrum, we run it through the stochastic approximation,

Â approximation module and then we can synthesize.

Â And we can synthesize the stochastical by, basically,

Â the idea is filtering white noise, but in the implementation is basically taking

Â the phases of random numbers and applying

Â 14:20

So let's now see an example of a complete analysis synthesis of a particular sound.

Â So we're taking this saxophone sound, let's listen to that.

Â [MUSIC]

Â Okay, and then below it we have the two representations that we have obtain

Â the harmonics and the stochastic component,

Â the spectragram of the stochastic component.

Â Let's listen to the harmonics.

Â [MUSIC]

Â Now we may not appreciate what is missing but

Â when we listen to this stochastic approximation, [NOISE] we have

Â to make it a little bit louder in order to actually listen what is going on.

Â Well, with these two components,

Â we basically have analyzed and modeled the original signal and

Â we can put them together and generate this synthesized sound,

Â [MUSIC]

Â That captures most of what is perceptually relevant in the sound.

Â So, for these topics that I discuss this lecture,

Â there is not that many references in terms of tutorial or

Â sort of more introductory material.

Â But, there is quite a bit of articles that have been proposing

Â different strategies to analyze sinusoids,

Â obtain residuals, approximate the residuals, etc.

Â So in this link that I put here on the website of the MTG,

Â I have kept some articles, well quite a bit of articles

Â that have been published related to these issues.

Â So feel free to go there and you can sort of find those articles.

Â And that's all basically in this lecture.

Â We have covered the most advanced models that we will be presenting in this course.

Â We basically combine all the previous models,

Â developing a variety of analysis and

Â synthesis techniques that can be applied to many sounds and for many applications.

Â In the next lecture,

Â we'll focus on how these models can be used to transform sounds.

Â So I think we're going to start having fun in doing some interesting new sounds.

Â So I hope to see you then, bye bye.

Â