0:00

Hi. Welcome back. In this set of lectures we're talking about Markov models. And in

Â the previous lecture I introduced what they were. How they are these finite set

Â of states and there's transition probabilities between those states. What I

Â want to do in this lecture, is just show you how a simple Markov process works. So

Â we're gonna take a very, very simple Markov process and work through it. See

Â exactly how dynamics unfold. And in doing so we're going to learn how to use

Â matrices. How to actually, specifically how to multiply matrices. Okay, so let's

Â take our simple example, and let's do the case of the alert and bored students. So

Â I'm teaching a class, like many people do. This is an online class. There's some

Â percentage of people that are alert, and there's some percentage of people that are

Â bored. And what's going to happen, at any given moment in time, someone whose alert

Â could switch and become bored, and someone who's bored could switch and become alert.

Â And what we want to do is we want to understand the dynamics of that process. We want a

Â model that helps us understand these dynamic processes where there's these

Â states, alert and bored, and people move between them. So we got to make some

Â assumptions. We've got the two states, alert and bored. Now we need

Â to understand the transition probabilities. We need to assume something

Â about transition probabilities. So let's assume the following is true: that 20

Â percent of the alert students become bored, but the 25 percent of the bored

Â students suddenly say, "hey, that sounds interesting, these Markov processes sound

Â really cool", and they become alert. So, that's what we wanna model. So we can

Â think of that as, we've got this set of alert students. 20 percent of them are

Â gonna become bored. And of the bored students, 25 percent are gonna become

Â alert. So we can draw this picture, but this doesn't help us much. We wanna sort of

Â figure out what's gonna happen. And this is where the matrices will be useful. Now,

Â before we do the matrices, let's just try to do it by hand. So, let's start with

Â this scenario. You've got 100 alert students. So, if I have 100 alert. Let's

Â call this A, is 100. And bored is zero. And I know that 20 percent of

Â the alert become bored and the rest are going to stay alert. So that means what's

Â going to happen is that I have 80 alert and 20 bored. We're now going to

Â think, okay, what happens next? What happens next, I know that of the 80 alert,

Â I know that 20%, which is 16, are going to become bored. And the rest, 64,

Â will be alert. Now the 20 that are bored, I know that 25%, 25% of that

Â is 5, so I'm going to put 5 of those, become alert and I'm gonna know

Â that 75 percent of it, which is 15, stay bored. So what I'm gonna get

Â is 69 alert and 31 bored. And I can say, okay now I've got alert, 69, and bored,

Â I've got 31. And now I've gotta do this again. I've gotta think, okay, well,

Â 20% of these, which is gonna be 13.8, become bored, and so on. And this,

Â you think, okay, this gets really complicated. And I've got numbers all over

Â the place. Maybe there's a better way. Maybe there's, instead of writing all

Â these numbers with these zeroes, there must be a simpler way to keep track of all

Â this. Well, there is, and the idea is something called a Markov Transition

Â Matrix. And here's the idea. We basically write down the probabilities of moving

Â from state to state. So, these columns tell us what's true at time t. And the

Â rows tells us what's true at time t+1. So if you're alert at time t, there's an 80%

Â chance you stay alert and a 20% chance you become bored. If you're

Â bored at time t, there's a 25% chance you become alert and a 75%

Â chance that you stay bored. So this gives us the matrix representations, simple

Â representation of all these transition probabilities. Now the reason this is

Â useful is, then we can just multiply these matrices to see how the transitions

Â unfold. So, here's an example. Suppose I start with 100%, or one. This is just a

Â probability, so probability one summons alert. And I want to figure out how many

Â alert people are next time. This number, this point eight, tells me the percentage

Â of alert people that stay alert. So I'm going to get, so I take point eight and I

Â multiply it by the one. That gives me point eight. And I want to ask, how many

Â bored people become alert? Well, 25 percent do. And how many bored people were

Â there? Zero. So I'm going to get point 25. Times zero, so I end up with point eight.

Â So the way I multiply matrices is I basically take this row here, and I

Â multiply it by this column. Now, let's make this more formal. So, what I do to

Â multiply these things out is, I've got here's where people go at times t+1.

Â This is how many are gonna be alert at time t+1. 80 percent of the

Â alert people, and 25 percent of the bored people. Here is the percentage of alert

Â people, and the percentage of the bored people. So I want to know how many are

Â going to be alert next time. 80 percent of the one, and 25 percent of the zero, and

Â that's going to mean point eight. Now I want to ask how many are going to be

Â bored. Well, 20 percent of the alert people, and 75 percent of the bored

Â people. So I take this row, and also multiply by the composite. Take point two

Â times one, and point 75 times zero, and I get point two. So what you get is, when I

Â multiply this matrix, the transition matrix here by this initial

Â vector one, zero. I get 80, 20 just like I did before. Remember the way I did

Â that is I take this row, multiplied by the column and then I take this row,

Â multiplied by the column. Watch now, I can do it again. Now I'm at 80, 20. And I

Â wanna ask how many am I gonna get next period. Well I take this row 80, 25 and

Â multiply it and see 80 percent of the people are alert, 80 percent of them stay

Â alert. So that's right there, 20 percent are bored, 25 percent

Â become alert. So that's right there. And when I add those up, I'm going to get 69.

Â And here I'll get 31. We'll see what happens in the next period. I again just

Â take this row times this column. So of the 69 percent that are alert, 80 percent of

Â them stay alert. And of the 31 percent that are bored, 25 percent of them become

Â alert. And I get that 63 percent are alert. And then I can do it again. Take

Â point eight times the 63, point 25 times the point 37, then I get the new

Â percentage that's alert. Which is going to be 60%. And I can keep going, and going,

Â and going, and if I do it one more time, I end up with 58%. So what does that tell

Â us? That tells us that if we started with all alert students, after six periods, I'm

Â gonna end up with 58 percent of students being alert. Now we wanna know, where does

Â this process stop? Is it going to go end up with nobody being alert? Well, let's

Â think that through. Let's suppose that we started out with nobody being alert. And

Â we can ask what happened. So I started with all bored students. What's gonna

Â happen? Well now all I do is put a zero for the alert and a one for the bored and

Â ask what happens next. Well, 80 percent of these alert students will stay alert, but

Â that's zero. So it's .8 times zero. And 25 percent of the bored students will become

Â alert. That means I'm gonna get .25. And that means, since this sums to one, I'm

Â gonna get .75 over here. Next period I've got: 25% of the students are

Â alert, and 75 percent are bored. Well, now I can just put that here as my population

Â at time 2, and I can think, okay, of these 25% that are bored, or alert,

Â I'm sorry, 80 percent will stay alert. Of the 75 percent that are bored, 25 percent

Â become alert. And if I multiply all that out, I get that 45 percent are alert, 55

Â percent are bored. If I do it again, I'll end up with an even number of alert and

Â bored. And if I do it one more time, I'll get that 53 percent are alert, and 47

Â percent are bored. So it sorta looks like this is going to an equilibrium. When I

Â started off with everybody alert, I got down to 58 percent alert. And when I

Â started out with everybody being bored, I ended up with 52 and a half percent being

Â alert. So it looks like it's converging somewhere between 53 and 58%. Well, how do

Â I find what that equilibrium is? This is where the matrices become really powerful.

Â So, let's think of it this way. There's some percentage of people that are alert.

Â That's p. There's some percentage of people that are bored, that's 1-p. What

Â I'd like is, after this process takes place, for the same percentage to be

Â alert. So how many people are going to be alert? Well, that's going to be .8 p plus

Â .25 times one minus p. So what would it mean for there to be an equilibrium? The

Â equilibrium would mean that after I multiply this out, I have the same

Â percentage of people being alert. So the equilibrium, I've put a little star here,

Â is gonna be the p-star, actually .8 p-star plus .25 times one minus p-star. Just

Â gives me p-star back. That I end up with the same percentage of people alert that I

Â started. This just becomes algebra. I can now write this out. I've got an equation

Â where this is my Markov transition matrix. And I want some probabilities p of people

Â being alert. So I should after the transition, I've got p back. Right? So

Â that's just gonna be .8 p plus .25 times one minus p, should

Â equal p. I wanna find the p that solves this. Well, let's multiply through by

Â 20, just to make this simpler. So I'm gonna get 16 p plus five times one

Â minus p equals 20 p. So that's going to give me 16 p plus five minus 5 p equals 20 p.

Â And so we bring all the "p"s over to the one side, and we get five equals 9 p, so p

Â equals 5/9. So what that says is that if I take with 5/9 of the people being

Â alert, I'm going to end up with 5/9 of the people being alert. So let's think

Â about how that works precisely. So 5/9 of the people alert. What do we know? We

Â know that 20% of them are gonna become bored. So that means 20%

Â means that each period, one-ninth of the population will become,

Â move from alert to bored. I also know that 25% of the bored people will

Â become alert. What's 25% of 4/9? That's also 1/9. All

Â right? So what I, is that, so each, each period, 1/9 of the people are moving

Â from alert to bored, and 1/9 of the people are moving from bored to alert,

Â which means that exactly 5/9 stay alert and exactly 4/9 stay bored. Now notice

Â what this equilibrium is like. It's a statistical equilibrium. So if we can

Â think of an equilibrium point where nothing changes, here the thing that's not

Â changing is the probability. So the population is still churning. People are

Â moving from alert to bored. But if I think in terms of probabilities, that

Â probability is staying fixed. That probability is staying fixed

Â at 5/9. People are moving around, but the probability's staying fixed.

Â That's why this is sometimes called a statistical equilibrium, 'cause the

Â statistic p, the probability of someone being alert, is the thing that doesn't

Â change. Okay, pretty involved, right? What we did is, we wrote down the Markov

Â transition matrix. And we showed how using that matrix, we could solve for

Â an equilibrium. And we saw, at least in the simple example of alert and bored

Â students, that the process went to an equilibrium, and it was fairly

Â straightforward to solve for. What we want to do next is we want to do [a] slightly more

Â sophisticated model that involves multiple states instead of just two, involves three

Â states, and we'll see how that process also converges to an equilibrium. Thanks.

Â