0:00

Welcome back to the course.

Â [INAUDIBLE] processing for music applications.

Â In the previous programming class, we talked about the peaks detection,

Â the fundamental frequency detection algorithm.

Â So how to identify the fundamental frequency of harmonic sound.

Â Now we are ready to put together the whole harmonic model analysis and synthesis.

Â So this is the block diagram of the system that we have and

Â we're going to explain, in which we start from the standard

Â blocks that we already have seen in the sinusoidal model.

Â So that it takes the inputs out in windows then it performs the DFT,

Â and then it finds the peaks, and these two blocks

Â are the ones that are specific for the harmonic model.

Â So the first one is the F0 detection, so

Â from the peaks, it identifies the best fundamental frequency.

Â 1:29

So let's go to the file harmonic model, the BY.

Â This is the file that includes most of the functions of

Â the harmonic model, the actual implementation of the x0

Â detection is in the utility functions file.

Â But here, we have the wrapper to the fundamental frequency at zero

Â detection that we already mentioned in the last lecture.

Â Then, we have the harmonic detection that we'll be talking about shortly and

Â then, we have the complete harmonic model, that has both analysis and

Â synthesis but this is meant to be run in real-time.

Â So it doesn't allow for cleaning of the tracks, so

Â we will be talking about today on the analysis and

Â synthesis separate because this allows us to do cleaning of tracks.

Â And so, handle little bit of memory of a past few frames and

Â this will result into better analysis and synthesis.

Â So let's talk about these function.

Â Harmonic model and all, so it inputs all the parameters of the analysis,

Â the input array of the samples, sampling rate.

Â The window, FFT size, hub size, the threshold, T for the peaks.

Â The number of harmonics that are allowed, so this is the maximum number.

Â Then the range of frequencies for the fundamental

Â frequency attraction, so this mean F0 and the max F0.

Â Then there is this error of threshold that is also part of the F0 detection

Â algorithm, so this is a minimum error that we'll be accepting.

Â And then, for the sort of a track continuation,

Â harmonic continuation, we have these two parameters.

Â One is the harmonic deviation slope,

Â this is sort of a deviation slope from the previous

Â 3:38

series of harmonics and from the ideal harmonic series.

Â And this is a way to account for that higher harmonics should allow for

Â bigger deviation than lower harmonics.

Â And then finally, parameter to control the length of harmonics tracks so that

Â we don't accept shorter tracks than the one's that specify with this parameter.

Â And the actual code is again, very similar to the sign of model,

Â we have a main loop that iterates over the whole sound and

Â it keeps analyzing the spectrum and in the peaks.

Â Interpolating those peaks and then here is where the specific code of

Â the harmonic models tabs, so we call the fundamental frequency algorithm.

Â Okay, so this is the one from the utility functions that

Â we talked about in the last lecture.

Â Then we have these few lines that make sure that this new fundamental

Â frequency is not that different from the previous one, so that it maintains

Â sort of continuity and then there is this function called harmonic detection.

Â This is the one that identifies the harmonics of fundamental

Â frequencies and it's above this code.

Â So let's go to here, this is the harmonic detection function.

Â Okay, so this is the one that given the fundamental frequency,

Â F0 and the peaks of the spectrum, it identifies which of these

Â peaks are close to multiples of this fundamental frequency.

Â We also as inputs, we accept some information from the previous strings so

Â that we can identify if there is some time continuity.

Â And then these deviation that is the deviation we allowed from

Â perfect harmonicity and from perfect continuity from the previous frame.

Â 5:52

Okay, so the code is not that complicated.

Â The core is basically these three lines In which

Â we defined the deviation from perfect harmonicity.

Â So we find the peaks compare or

Â a particular peak compared with a particular perfect harmonic.

Â We also compared a peak with respect to the previous harmonic

Â coming from the previous frame.

Â And then, we define the threshold base

Â on these harmonic deviation of slope that is the one that will be accepted.

Â So if the current peak is within that threshold therefore,

Â within the kind of vertical deviation and horizontal deviation,

Â so it has this frequency and time evolution tracking.

Â So we accept that peak as harmonic and

Â we keep adding to the harmonic that we create and that's all,

Â that's basically what the harmonic detection does.

Â So if we go back to the harmonic model now [COUGH] after this harmonic detection.

Â Let's say this is a setup some code to create the array of values that

Â keep adding up to the whole matrix of the output.

Â And finally, if once the whole process is done, so

Â once we have gone through the whole sound.

Â We clean the tracks, so

Â we delete the tracks that are smaller than a given duration,

Â and again, this is the same then with it in the sinusoidal analysis.

Â Okay, so I wrote a script that basically analyzes

Â a sound with this harmonic model and now function,

Â so we took this vignette sound and specified all the parameters.

Â So the number of parameters are starting to get quite a few.

Â So we have the window, we have the window size, the FFT size, the threshold.

Â There is no need for this comma here, so minus 90 as a threshold,

Â then minimum duration in time, the number of maximum number of harmonics.

Â The minimum fundamental frequency and

Â maximum fundamental frequency, the errors threshold that we accept and

Â then this deviation slope that we just talked about.

Â Then there is these two parameters that are normally fake because they

Â are more related to synthesis which is the FFT of the synthesis buffer.

Â And the hop size which are given that the FFT buffer requires for

Â the synthesis, it requires the hop size to be one-fourth of that.

Â So we normally put H as being 128, so

Â we normally don't allow any changes from that.

Â So from the graphical interface, there is no control over these two parameters.

Â Okay, and then we read the sound, we get the window and

Â we call this harmonic model an function.

Â And here, I just plot the frequencies of these trajectories, so let's run this.

Â So let's run test three, now let's first save it.

Â Okay, and now let's run it.

Â Okay and this is the trajectories we obtained,

Â okay they are okay, they're pretty good.

Â If we zoom in into a particular region,

Â we might be able to see a little more, okay?

Â Of course, if we zoom in we're starting to see

Â the problems at the transitions between nodes.

Â And especially, we see a problem as the harmonics go higher and higher up.

Â This is because this is a linear scale and

Â this deviations are kind of logarithmic.

Â So as the frequency goes up, in fact the changes

Â get much larger because they are logarithmic changes.

Â So in fact,

Â this is the reason why we need this deviation slope that we specified here.

Â So for example, if we reduce this deviation slope,

Â let's make instead of 0.01.

Â Let's make 0.001 and let's run it again.

Â 10:48

It does restrict that the deviation allowed in the higher frequencies.

Â So it has the increase of deviation is much smaller,

Â so it has not allowed these larger deviations.

Â So with that, we have quite a bit of control of how this

Â tracking process is implemented.

Â Okay, so in the SMS tools package, there is this harmonic

Â model function that performs both the analysis and synthesis.

Â And is in fact, the file that is called from the interface,

Â so in this file we have a one main function and

Â it has all these parameters that we already talked about.

Â But apart from the analysis that we already performed, it does the synthesis.

Â Okay, and then it saves the file into a synthesis file and

Â it plots analysis and synthesis of this.

Â So let's run this harmonic model function.

Â 12:02

Okay, also this would take a little bit longer because it has to do

Â the analysis and then the synthesis.

Â And then it just plots to the first 5,000 hertz and

Â specified to plot on the phase 5,000 hertz.

Â So here we see all these trajectories that evolved in time and

Â the synthesis at least the wave of loops are quite similar.

Â So let's listen to the output sound, so

Â now we created a file called bignish_harmonicmodel.wav.

Â So I can play it so, if I put the exclamation

Â mark play just say type vignesh.

Â [MUSIC]

Â Okay, so this is the synthesized version of the original file and it's quite good.

Â I mean, we will see starting next week that in fact there is some things

Â that are left that are not included in this resynthesis and

Â this is basically the residual component that We are missing.

Â But anyway, that's all for the harmonic model and we go back to the slides.

Â 13:42

definitely encourage you to go through that and run it and abuse it if you want.

Â So that was all for this programming classes on the harmonicModel.

Â Hopefully, this has given you sort of a depth,

Â in-depth view of what it entails to implement this type of models.

Â And well, we will continue, so next week this will get a little bit

Â more complicated because we will try to see what is missing out of this model.

Â And we will try to model this missing part,

Â this is going to be the concept of the residual or the stochastic components.

Â So I will see you next week, bye-bye.

Â Thank you.

Â