Hello, everyone.

In this lecture, we will try to estimate model parameters of

autoregressive processes of order 2 doing some simulations.

In others words, we are going to simulate an AR(2) process.

So, the objective is to estimate the variance

of the white noise in the AR(2) process.

And estimate the coefficient of the simulated AR(2)

process using the Yule-Walker equations in a matrix form.

AR(2) process with mean zero would be in this format without any constant,

and the Zt's are innovations, the white noise.

And we tried to simulate this process for phi 1,

the first coefficient being 1 over 3.

Phi2, the second coefficient is 1 over 2.

And sigma is 4.

In other words, this AR(2) model has three parameters.

So we're going to use the Yule-Walker equations.

And eventually Yule-Walker estimators to actually estimate each of

these coefficients, the phi1 and phi2 and also sigma in this problem.

So we estimate the coefficients of the model by first finding r1, r2.

Remember r1, r2 are sample auto correlation function.

We're going to use acf routine in r.

And we're going to solve the system using the following

matrix form of the Yule-Walker equations.

This is our r matrix which is symmetric, it has an inverse.

And then we're going to find the inverse of it and

multiply it to the left side which is r1, r2.

And we will get phi1.hat, phi2.hat.

But before we start the estimation, let me mention something about the sigma.

So, we know how to estimate the coefficient in the AR(2) model but

how about the sigma?

That's also another parameter of the system so

we should be able to estimate that as well.

So let me note the following.

If you look at this system, we assume that before we start with the Yule-Walker

equation, we already assume that it is a stationary AR(2) process.

So this is a stationary AR(2) model, AR(2) process.

We take the variants from both sides.

We get the variants of Xt, variants of Xt minus 1, variants of Xt minus 2.

Which all of them are the same because we have weak stationary or

covariant stationary process.

But since Xt minus 1, Xt minus 2 have some correlation, we also have

this covariance Xt minu 1, Xt minus2 term here and the variance of Zt is sigma.

So we're going to use this equation to get the sigma.

So sigma will be equal to the variance of this process

is actually gamma 0 autocovariance at lag.

And then we have 1 minus phi1 squared phi2 squared.

But if I pull out variance from here this covariance,

this is of the autocovariance at lag1, this is gamma 1.

If you plot variance you will get gamma 1 over gamma 0 which is actually rho 1.

So this is basically how we can estimate sigma.

But we can actually simplify this a little bit more.

Realize the following, from the Yule-Walker equations in the matrix form,

we realized that rho 1 is actually from the matrix multiplication is equal to

phi1 plus rho1, phi 2.

In a similar way, rho2, is the same as phi1, rho1 plus phi2.

This comes from the Yule-Walker equations.

Then, I take this expression which is inside this sigma expression.

Sigma is equal to gamma 0 times that.

Now here, we can separate this 2, phi1 phi2, rho1 expression into two of itself.

Of phi1 phi2 rho1, phi1 phi2 rho1.

In these two terms, we pull out phi1.

In these two terms we pull up phi2 and

we realize that the parenthesis is actually rho1 and rho 2.

In other words, we actually get sigma.

We get a formula for sigma square.

Which is gamma 0 autocovariance of the system at lag0.

The variance of Xt times 1 minus phi1 rho minus phi2 rho2.

Now, we can estimate sigma from here.

And this is going to be called the Yule-Walker estimator, instead of gamma 0,

we're going to get C0.

This is a sample of autocovariances at lag0.

We are going to use ACF routine to get this.

And we'll find phi1 and phi2, these are our coefficients which will come

from the Yule-Walker equation in the matrix form.

And we're going to multiply them with r1 or r2,

which are sample of the correlation function.

And this will give us an estimate for sigma squared.

So let me mention the following.

Now I'm going into the details of the simulation.

And I'm going to open up the code.

Code is written in Notebook.

So at this point, if you haven't all ready, stop the video.

Go and open up the Notebook.

Which is called AR (2) simulation.

And we're going to carry out every step that I'm talking about in this

presentation.

Actually it's written already in that notebook.

And try to run every code block in Notebook.

So, let me first go over this and then I'll open it up as well.

Number of data points, we're going to choose 10,000 data points.

And we're going to use a few routines here arima.sim, this is the simulation.

So arima.sim simulates data from the arima models.

Well, what is arima?

This is auto regressive integrated moving average models.

We have about moving average models.

We have talked about auto regressive models.

But we haven't connected them yet but

we will still use this model to get to the AR model.

We're going to use plot routine for plotting.

We're going to use acf() to find out the correlation function.

We will also use that to find the autocovariance function.

And we have done this before.

Matrix, this basically defines the matrix with dimension m and n.

And we're going to use solve(R,b).

Where R is a matrix, b is a vector.

And this routine solves Rx=b and gives a solution.

Sigma, we're going to choose 4.

And phi[1:2], this is phi1 and phi2.

It is defined with this array, 1 over 3, 1 over 2.

N is 10,000.

We will set.seed(2017),

so that we will both get the same dataset.

So you can reproduce exactly what I'm reproducing in this lecture.

And ar.process, we are going to use arima simulation.

This is the model, model takes the list.

And then if you write ar, this specifies the coefficients of the ar model.

And standard deviation is equal to 4.

Basically specifies standard deviation of the innovations, the random noise.

And if you do that, since you have set the seed(2017),

you should get the exact same ar.process.

And if you, for example, look at the first five elements

out of 10,000 points in this time series, you should get the following numbers.

Then, what do I need?

I need r's in my Yule-Walker equations.

I'm going to get my r's [1,2] or R1 and R2.

ACF gives you sample auto correlation function for a few lags.

If I take this $acf which means I'm going to the acf part only.

And I'm going to look at acf2 and acf3,

because the first element in this array is actually 1.

That's actually row 0.

Okay.

So if you look at this, r[1] and r[2] are going to be following numbers.

I'm going to define R as a matrix, [1,2,2].

What does it mean?

We want to have a 2 by 2 matrix, with all elements 1.

So we get this R.

But, how do I do that so that I can actually edit this?

So I'm going to edit 1, 2, this term and this term, by r1.

This is the matrix R in AR(2) process.

Let's define matrix b, which is the right hand side which is r1 and r2.

So we put them side by side.

And we try to solve (R,b).

Because R now we have it, b now that we have it.

We can solve it.

Once we solve it in R, R gives us the following answers.

The first one is an estimate for phi1.

The second one is an estimate for phi2.

So we called them phi1.hat and phi2.hat.

And we can actually put them together here.

This is an array to see phi1 and phi2.hat.

And I define this as a matrix, 2 by 1 matrix.

And then phi1.hat becomes that matrix.

Just define c0 as an autocovariance at lag0.

That's acf1, because the index of R always starts at 1.

So we have to do acf1 here.

And then type to do covariance so that it doesn't find out the correlation function,

it finds out the covariance function.

And we calculate the var.hat.

Var is for variance, so we are trying to estimate the variance.

Well, we just did it.

This is c0 times 1 minus phi1 r1, phi2 r2.

Basically the product of phi1.hat with r.

And then we're going to partition our screen,

our output device, into three rows and one column.

In the first row we plot the process with this title.

In the second row, we're going to have acf, autocorrelation function.

And in the third row, we're going to have a partial autocorrelation function.

Okay, so this is the Notebook.

We have the name, which is Yule-Walker Estimation- AR(2) Simulation.

This is the name of the file Simulation of AR(2) process.

We are simulating AR(2) process.

Let's set seed the common number, so that we can produce the same dataset.

So let me run this cell, okay?

Now, let's look at the model parameters, we're going to say sigma is 4.

And this is phi equal to null.

Now let's define phi, phi is basically 1 over 3 1 over 2, phi1 and phi2.

Let's just print it and just to make sure to check that we have the right phi.

Exactly this is 1 over 3, this is 1 over 2.

Okay, let's run.

And n equals 10,000.

All right and we come to a simulation.

We simulate using arima simulation.

First entry in this function is n, which is 10,000, right?

So we are simulating a time series with 10,000 points, what is the model?

Model has to be a list.

We have to give the AR,

in author arrays of coefficients which is 1 over 3, 1 over 2.

We can actually write here phi1, phi2 if you like.

And we have to make sure that the standard deviation of the innovations,

the random noise is 4.

And the standard deviation is equals to 4.

And let's make sure we have that right AR process.

Let's run this and we get the first five entries in the time series.

The first five data points.

Here we are trying to find a name, the 2nd and

3rd sample autocorrelation function, right.

In other words, R is null, we define R.

R1,2 is going to be ACF.

We do not want to plot so we say plot is false.

But we have to take second and third term because the first term is actually rho0.

So first term is always 1.

We want to get r1 to r2,

we have to look at the second and third entries in the ACF.

Okay, if we do that, we get r1 here, this is r1.

Autocorrelation coefficient in lag1, autocorrelation coefficient at lag2.

And let's define matrix R.

So matrix R is going to be a two by two matrix, every element is 1.

Here we go.

This is our r, every element is 1.

And if I look at this cell, I am editing first row,

second column and second row, first column.

This other not the main diagonal but other diagonal by r1.

So if you do that, run the cell.

We will update the following R.

This is exactly what we were looking for.

This is the R that's coming from the Yule-Walker equation.

Let's define R matrix b which is on the left-hand side.

And that's basically r1 and r2.

Okay, now its time to find the coefficients, phi1.hat phi2.hat.

All we have to do, we have to solve Rx=b and how do we do this?

We say, solve(R, b), right?

Solve(R, b) will give us a phi1 and phi2 together.

But if I look at the first element, that's phi1.

If I look at the second element of that matrix, that is phi2.

And I take this phi1 and phi2 and put it into the matrix, phi1.hat.

This is matrix with two rows and one column.

And if we do that we get the following.

So the first estimate here is for phi1.hat.

This is phi2.hat.

Okay.

So we're almost done, right?

We have the AR(2) process.

We have phi1 and phi2.

Note the following, the original phi1 was 0.333.

This is 0.34 close, but not really close.

And here we have phi2.

The original phi2 is 0.5.

Estimation gives us 0.48, not bad.

We can get a better estimate by increasing the number of data points.

Let's look at the variance because this is the only thing that's remaining,

is to estimate the variance of the innovations.

So we take the autocorrelation function, but we say type covariance, right?

We will get autocovariance at lag0, and

we have to start from 1 because of the indexing in the r.

And then rho.hat is defined to be autocovariance, this is gamma 0,

estimation of the gamma 0 c0 times 1 minus that product of our coefficients and

our autocorrelation coefficients.

And then we estimate this we get var.hat which is the variance,

the estimation of the variance which is 16.37.

Remember the original variance was 16, the square of 4, What we get here is 16.37.

And if we plot, so if we partition the output into three rows,

one column and we plot the time series acf and pacf.

And if we do that we obtain the following plot.

This is our time series.

This is the ACF and this is a PACF.

Just note the following, this abuses the time plot, but if you look at the ACF.

The ACF is decaying, eventually, right?

This is very much like the typical AR process.

And if you look at the partial ACF, partial autocorrelation function,

there are only two significant partial autocorrelation coeffitents at lag1,

at lag2.

And then as if there's no autocorrelation, that is very typical.

Because, PACF of ARP process has to cut from lagP.

Here we are looking at the AR(2) process.

So we shouldn't expect to see anything after lag2.

So what are the results?

Result is that phi1 is estimated by 0.34,

phi2 is estimated by 0.48, sigma is estimated by 16.37.

This was the actual model.

And this is the fitted model.

And this is basically what we just talked about, the time plot, ACF and PACF.

So what have we learned?

We have learned estimating model parameters,

in other words coefficients and standard deviation or the variance of

the innovations of a simulated autoregressive processes of order 2.

And we did this by using Yule-Walker equations in a matrix form.

Yule Walker Estimation - AR(2) Simulation

Practical Time Series Analysis

Meet the Instructors