sineModel

Loading...

From the course by Universitat Pompeu Fabra of Barcelona

Audio Signal Processing for Music Applications

196 ratings

Universitat Pompeu Fabra of Barcelona

Audio Signal Processing for Music Applications

196 ratings

In this course you will learn about audio signal processing methodologies that are specific for music and of use in real applications. We focus on the spectral processing techniques of relevance for the description and transformation of sounds, developing the basic theoretical and practical knowledge with which to analyze, synthesize, transform and describe audio signals in the context of music applications. The course is based on open software and content. The demonstrations and programming exercises are done using Python under Ubuntu, and the references and materials for the course come from open online repositories. We are also distributing with open licenses the software and materials developed for the course.

From the lesson

Sinusoidal model

Sinusoidal model equation; sinewaves in a spectrum; sinewaves as spectral peaks; time-varying sinewaves in spectrogram; sinusoidal synthesis. Demonstration of the sinusoidal model interface of the sms-tools package and its use in the analysis and synthesis of sounds. Implementation of the detection of spectral peaks and of the sinusoidal synthesis using Python and presentation of the sineModel functions from the sms-tools package, explaining how to use them.

Peak detection14:49

Sinusoidal synthesis12:48

Meet the Instructors

Xavier Serra
Full Professor
Dept. of Information and Communication Technologies, UPF
Prof Julius O Smith, III
Professor of Music and (by courtesy) Electrical Engineering
CCRMA

Welcome back to the course in original processing for musical applications.

This week are talking about the sinusoidal model.

In the programming lectures we have been trying to put together a whole system for

analyzing and synthesizing a sound using the sinusoidal model.

In the first programming lecture we talked about the peak detection, and

how to detect the peaks of a spectrum, which hopefully are going to be sinusoids.

Then in the last programming lecture, we talked about how to synthesize

sinusoids from those values, from the big values.

And now, in this lecture, we want to put it all together and

that the missing components in order to actually analyze a complete sound.

So we start from signal of x, and then we do the standard windowing FFT and

these are the things that we talk about this week.

So the peak detection and how to get the amplitude, frequency, and

phases of the peaks.

Then today were going to talk about the sinusoidal tracking, so

how to build a tracks out of the peaks.

Then, once we have this sinusoidal track we can do the synthesis,

so we can synthesize the spectral peak.

So with the shape of the sinusoids in the spectral domain,

and then we can do the inverse FFT and do the overlap-add so

that we can add the time-bearing aspect of the sound and

we can reconstruct very complex sounds.

Let's open up the file that includes most of the functions

that are needed in the sinusoidal analysis,

which is the sineModel.py which is part of the SMS tools package.

In this file, we have several functions that are of use for

the sinusoidal modeling.

For example, this first one is the sinusoidal tracking, then there is

another one which is cleaning sine tracks, that we will talk about.

Then there is one that implements the whole sinusoidal model analysis and

synthesis, but does not include the tracking aspect,

because this is meant to be used In real time.

So it cannot do a tracking and allow some memory in the process.

So we do not recommend to use that unless you want to

develop some real time kind of concept.

So what we're going to be using, and we recommend to use,

is the sinusoidal sineModelAnal function.

That just performs the analysis part and

is able to do sinusoidal tracking, because it allows to have

all the tracks in one place, and then do some post processing of them.

Okay, so in here the input is the whole inputs out x,

the sampling rate, the window, the FFT size,

the hop size, then the threshold for the pic detection.

And then a number of parameters that are important to control

the time-bearing aspects of the sinusoidal tracking.

For example,

one is the maximum number of sine that will be allowed at the same time.

Default is 100, so normally you put quite a lot of them, so

to allow for creating and disappearing sine wave.

And then we control the minimal duration of a track, so default is 0.01, so

the track that they're going to be shorter than this value are going to be deleted.

And then we going to control the allowed deviation

from one frame to the next of a given track.

So in this case, this deviation, which is in Hertz, 20 Hertz,

it means that in the lowest frequency, it allows from one frame to the next,

for a peak to change frequency by 20 Hertz and still be part of the same track.

And then this freqDevSlope is a way to allow for

this deviation to increase as the frequencies are higher.

In higher frequencies, normally, you would like this deviation to get higher.

So we can change that by controlling this slope.

The bigger it is, the bigger the slope, and therefore,

the bigger the change with the higher frequencies will be.

Okay, and then the function, it's not that long.

Of course, it has a lot of the parameters and

the functions that we have already seen.

The core is this while loop that basically,

what it does is, it iterates over the whole sound,

and it performs all the blocks that we talked about.

So it performs the DFT, using the FFT algorithm,

it performs the pic detection, then it performs peak interpolation,

and then it puts the speak into tracks.

And this is what we're talking about now.

So let's look into this function, the sinusoidal tracks,

which is right here in this function.

From the peaks that the algorithm has found in a particular frame,

and from the concept of tracks, the concept of a series of incoming tracks

with specific frequencies, we are building tracks.

We are basically continuing the incoming tracks, or

creating new tracks, according to these deviation parameters.

So, this function is a little bit long, but it's a lot of cleaning and

setting up things, the whole concept is not that complicated.

It's simply the idea that, if there are incoming tracks, and

if not, tracks will be created.

But, from these incoming tracks, it finds

the peaks that are closest to those tracks.

And then, if it's close enough,

according to deviation, they will become part of the track.

So, if this deviation is small, we will add it,

and otherwise, we can also create new tracks here.

Okay, so if there are peaks that are left that have

not been assigned to any existing track, we can create new tracks.

So tracks will be created and will be disappearing

if they are not used in the following frame, okay?

So this returns a series of tracks.

Okay, so if we go back to the function of the sineModelAnal, okay?

After this tracking is done, again, there is a whole

bunch of variables needed to handle these tracks, and

how to pack them in and clean the matrices so that we save space.

But then this is done, until then, then once everything is done,

before returning the tracks, there is a cleaning step.

So this cleaning step receives all the tracks that have been completed for

the whole sound, and

it will delete the ones that are shorter than what we specify.

So, if we go to this function, the cleaning of the tracks,

cleaning of sine tracks, it's very simple.

It just simply goes through all the tracks that have been created,

it looks for the beginning and endings of each track fragment.

Let's say, and if a track fragment is below a given

length then it will simply delete the track,

it will set the frequencies to 0 and

that's what it means to clean the tracks.

Okay, and that's all the sinusoidal model does, the analysis of that.

So let's use a function or a script that calls the sinusoidal model a noun.

So this a script that calls the sinusoidal model and out.

And with the given set of parameters with the sound,

over sound, with a different window, given a different size etc.

All of the parameters that we have talked about.

So let's run this function, or this script,

and well, it will print the actual tracks.

Let's run this then, run test6.

Okay, so this of course,

will take a little bit more than a single short-time Fourier transform.

So this is the result.

Here we have all the tracks and we see the tracks in different colors.

And things that are important to notice here is that, for example,

even if we see a single line, we can see that there are several colors.

That means that a track was alive for some time,

and then it disappeared, so a new track started and then it disappear.

A new track started and disappeared, etc., etc.

So this is the idea that tracks can appear and disappear.

Then also we see that most of the tracks correspond

to the harmonics of the sound, but some do not.

For example, hearing the high frequencies.

Clearly, we see tracks that are quite short lived and they may be tracking some

noise component, or they may be tracking some side note of the analysis.

And this is something that with the sine model we cannot do anything about it.

Of course, we can control the length of all these tracks,

the minimum length was the one that we established, which was 0.1 seconds.

So all the tracks here at least are 0.1 seconds.

If we had set this up to a different value, for example, let's leave it to 0.0,

so now we can run it again,

it will do the same analysis,

but it will allow to do many more tracks.

So now we'll track things higher up,

but if we zoom into the area we were looking at,

we see many tracks that are very short and presumably,

correspond to fragments of little sounds, or sight lopes, or things like that.

So the cleaning of the tracks is a good way to get rid of all these kind of,

that typically, maybe, corresponding to artifacts.

Okay, then we can do the synthesis, so

the sine model synthesis is this function here.

And it will receive these tracks that we analyze, and

from then it will call, basically, the generation of the spectral sines.

So it will iterate over all of them, and

it will generate the main loads of the windows.

Here there is some consideration for the phase.

We might want to not pass any phase, here it passes the analysis phase.

But in some cases, the phases will be reconstructed from scratch.

And it allows for reconstruction of the phase at the synthesis stage.

Anyway, so this sign model

function which is in the actual function that is

called by the interface that we use in the demo class.

It has one function, main, that calls the analysis and

synthesis of a given sound using the same sort of model.

So the parameters, the default parameters, it uses these bendier sound and

then it does analysis and

then performs synthesis, and it outputs the sound file.

And it also plots the input sound, the sinusoidal tracks and the output sound.

So let's just call these function.

Let's close this one first.

And if we run sine model function.

So we copied in our directory and

we had to change a few of the relative paths of files.

Okay, so now it's completing the analysis and synthesis of this bendier sound.

Okay, so this is the original sound, and

here we see these trajectories, and the resynthesis from that.

So if we zoom into here, for example, let's zoom into a particular area,

we will see all these quite messy sinusoids that are coming in and out.

And in this type of sound, that's quite normal,

because there is a lot of very partials that are very unstable and

they disappear and appear in a short fragment.

Okay, so we can listen to those sounds.

For example, let's listen to the resulting sound we created a file in this

directory the synthesize, and its called the NDR sine model assigned model.

So we can use the play a system command to play the sound.

[SOUND] Okay, and it's not that different from

the original file that the Bandera we input it.

So anyway, so that gives you an idea of how to use

the SMS tools code and how to understand it.

So now the idea is that by yourself you can play around with these parameters and

really go into the different functions and understand what they do.

So let's finish with that and, so we have seen the whole sinusoidal

model from a programming perspective and within the SMS tools package.

We have seen all the functions that perform all these different analysis tasks

and synthesis tasks.

So we have a pretty good code base

with which we can do a lot of very interesting things.

Again, this is just one step into further complication on these models.

So now we have seen the sinusoidal model as a model that will allow us to do

quite a few things.

But we can do more, and so next week,

we will be talking about the harmonic model, which is a way to restrict

the sinusoidal model to track only the harmonics like in the oboe sound.

I think it was clear that it makes sense to do that.

And we will also extend it with other new approaches to

the spectroanalysis and synthesis sound.

So I hope to see you next week, and thank you for your attention.

Explore our Catalog

Join for free and get personalized recommendations, updates and offers.

Coursera provides universal access to the world’s best education, partnering with top universities and organizations to offer courses online.

© 2019 Coursera Inc. All rights reserved.

Download on the App Store

Get it on Google Play