Learn fundamental concepts in data analysis and statistical inference, focusing on one and two independent samples.

Loading...

From the course by Johns Hopkins University

Mathematical Biostatistics Boot Camp 2

41 ratings

Learn fundamental concepts in data analysis and statistical inference, focusing on one and two independent samples.

From the lesson

Hypothesis Testing

In this module, you'll get an introduction to hypothesis testing, a core concept in statistics. We'll cover hypothesis testing for basic one and two group settings as well as power. After you've watched the videos and tried the homework, take a stab at the quiz.

- Brian Caffo, PhDProfessor, Biostatistics

Bloomberg School of Public Health

Okay. So the calculation when mu is, when

Â your alternative is less than the null value is very similar.

Â You can, if you're taking this class, you should be able to do it.

Â For the unequal to hypothesis, things change a

Â little bit, and here's the procedure I'm going to recommend.

Â pick one of the two sides.

Â Let's say, you know, in general,

Â it's probably easier maybe to do the bigger side,

Â say, mu, mu a being bigger than mu 0.

Â So pick the bigger one sided hypothesis with a,

Â you have to pick a, you know, again, to

Â do a power calculation you have to pick the

Â value of the, under the alternative that you're going to check.

Â And so let's assume you pick a value of mu a that's larger.

Â So calculate the power for the one sided hypothesis,

Â the for the test of a larger mu a.

Â Calculate that power, but using alpha over 2 as

Â the error rate instead of alpha for the error rate.

Â And this is right enough that it excludes

Â a little bit of probability in the power calculation.

Â But it's kind of not a probability that you

Â have to worry about so much, because it's so small.

Â And then well, basically, what it, what it omits is the probability,

Â if your alternative mean is larger than the null mean, the probability of randomly

Â getting a test statistic is so small that you reject in the other direction.

Â And, you know, so that probability is usually very small so who cares.

Â so you can just omit it.

Â Fancy power calculators might include it, but it the, the

Â usually that probability is so small it's irrelevant so I should

Â quit talking then.

Â as alpha gets larger, then the power goes up, right?

Â If you're requiring less evidence to reject the

Â null hypothesis, then you're going to detect more alternative hypotheses.

Â Just like in a court of law, if you if you don't require

Â much evidence to convict people, you're going to convict a lot of guilty people.

Â you, you'll also convict some innocent more innocent ones, too.

Â but, but that's not, you know, power is

Â only concerned with the probability of under the alternative.

Â power of a one sided test is obviously bigger

Â than the power of a of a two sided test.

Â and you can see this in the calculation pretty

Â easily in the way that I'm suggesting you do it.

Â Because when you calculate a quantile of alpha over 2, you move further out,

Â say, into the normal probability tail.

Â And then you know, the, the, the, the that value gets a little bit

Â bigger and the, you can see that the probability gets a little bit smaller.

Â let's see.

Â And then, the further your alternative mean gets away from your

Â null mean, either, you know, larger, if you're testing larger than

Â or smaller if you're testing less than, or absolute value gets

Â further away from mu 0 if you're testing not equal to.

Â then the farther it gets away, the greater the power is.

Â And that makes, of course, that makes total

Â sense in that you know, the, the, the,

Â the stronger the alternative hypothesis is, the easier it should be to detect it.

Â And then, of course, as your sample size goes up, your power, your power goes up.

Â And then, and then this is why sample

Â size calculations are so important, it's because, you know,

Â of all the things we're discussing, the one

Â that's only potentially under your control is usually n.

Â So, let's talk about calculating power for T

Â tests because things get a little bit more complicated.

Â So, so the power, just like we calculated,

Â well, let's assume it's a a greater than alternative.

Â The power is the probability that our test statistic lies in the rejection region.

Â So X bar minus 30 in our RDI Respiratory Disturbance Index example,

Â X bar minus 30 over S over square root n is bigger than our t quantile.

Â Now it's t with 1 minus alpha and n minus

Â 1 degrees of freedom with this calculation being executed

Â under the alternative hypothesis that mu equals mu a.

Â So now, if we were to do the same trick we

Â did before, where we were to add and subtract mu a,

Â and drag the term onto the other side, we'd still have

Â an S, which is random to contend with on the right-hand side.

Â so, that's, you know, that's a problem that, that doesn't

Â yield a collection of things that we know that we can solve for then.

Â so let's manipulate it around, and see what we can, we can figure out.

Â So this probability here, this top probability

Â is equal to the probability square root n X bar minus

Â 30 is greater than this t times S. And then we could divide

Â the left-hand side by sigma and the, the right-hand side by, by sigma as well.

Â Okay.

Â Let's talk about how we can do this calculation

Â given the tools that we have from the class.

Â So here, I've added and

Â subtracted a mu a, so we get a z random variable

Â here for square root n X bar minus mu over sigma.

Â Here we get something that we, we know or

Â we, we, we need to plug in values at least.

Â So, we get mu a minus 30 over sigma square root times square root n.

Â here, you know, again, to, to remember to do any power

Â calculation, you're going to have to plug in the value under the alternative.

Â so even though we don't technically know mu a, we've got

Â to be able to plug in potential values of mu a.

Â We, of course, know our t quantile in the square root n minus 1.

Â And then over here on this right-hand side just in case you didn't

Â notice it, I multiplied and divided by a square root n minus 1.

Â So this far most right-hand quantity, n minus 1 s squared over sigma squared.

Â Well,

Â remember if the data are iid Gaussian, that that works out to be a chi-squared

Â random variable so we have over here this square root of a chi-squared.

Â So I've just re-written this equation now down here on the second line, where I've

Â replaced the Z random variable by Z and

Â the chi-squared random variable by a chi-squared notation.

Â And we've stated it, we've never proved it.

Â We've stated it, that the Z and

Â the chi-squared are independent.

Â So this probability statement is a little bit

Â hard because it involves the bivariate distribution of

Â a Z and a chi, and a chi-squared, which is a little bit tricky to work with.

Â But we could actually do this via Monte Carlo very easily.

Â We could simulate pairs of Z random variables and

Â chi-squared random variables, and evaluate this inequality for every pair.

Â So we'd take a simulated Z random variable, we would subtract square root

Â n mu a minus 30 over sigma, I mean, sorry, we would add that.

Â And then we would take a chi-squared simulant and we would square root it,

Â and then multiply it by the t quantile divided by square root n minus 1.

Â And we'll just check to see which is bigger.

Â If the left-hand side was bigger, we would record a

Â 1 and if the right-hand side was bigger, we would

Â record a 0.

Â The percentage of ones from that simulation exercise

Â having generated lots of pairs of independent chi-squared

Â and Gaussians standard Gaussians, the percentage of ones

Â would then be an approximation to this probability.

Â And it would limit to this probability as we did more and more simulations.

Â So, if you wanted to do this more exactly you would have to be able to

Â numerically calculate the joint distribution of the normal

Â and the chi-square, which is not so hard.

Â you can figure out how to do it,

Â but r actually already has something, it's called power.t.test.

Â And here I give the r code just for

Â doing it for specific settings, and the result in this,

Â this case is a, is a power of, of 60%. So the function is power.t.test.

Â Of course, you have to enter in the n.

Â You have to enter in the delta, which in this

Â case is mu a minus the mu under the null hypothesis.

Â So let's assume that that's 2. and oh, a little fact that I, I should

Â have I should have alluded to before. If you look back at this

Â calculation here notice that the,

Â the, this formula here only depends on the

Â difference between mu a and mu 0 divided by sigma.

Â So we, so, so I, I, I was, I was actually

Â incorrect when I said we, we needed to know mu a.

Â We don't really, well, what, what, what we need to know is how different

Â mu a is from mu 0 in standard deviation units.

Â So, mu a minus mu 0 over sigma.

Â And notice mu a and mu 0 and sigma are all in the same units.

Â So this quantity, usually called an effect size is

Â is usually, what these problems are sort of characterizing

Â in terms of, because it's a unit free quantity

Â that has some hope of being interpretable across experiments.

Â so here we don't actually have to plug in a sigma we just

Â have to know what mu a minus mu 0 over, over sigma works out to be.

Â If we were to put in here that delta

Â equals 2, and then also add the argument that

Â sigma equals 4, we'd get the identical answer and

Â you can, you can try it to make sure.

Â Okay. So let's do the same exact calculation but

Â using Monte Carlo.

Â Here, I'm going to define the number of simulations, what is that, to a 100,000?

Â I'm going to set my n as 16, my sigma as 4, mu a, mu 0 to be 32 and 30.

Â And then I'm going to simulate a

Â 100,000 standard normals, a 100,000 independent chi-squares.

Â So now I have a 100,000 pairs of normals in chi-squared, and of course,

Â it doesn't matter how I matched them up, I could permute the normals or

Â the chi-squares and you'd still get the same thing.

Â here, my degrees of freedom is 15 because I have

Â 16 subjects and my degrees of freedom is n minus 1.

Â My t quantile is the 95th quantile of the percentile with the

Â t distribution with 15 degrees of freedom, and that's this qt function

Â right there. And then here I this line

Â right here ignore the mean statement for a second.

Â Imagine if I had just started with the z here

Â and said z plus square root n times mu a

Â minus mu 0 divided by sigma greater than t divided

Â by square root n minus 1 times square root x squared

Â [COUGH]

Â x squared being the random chi-squared.

Â That quantity would simply evaluate the pers the it would

Â return a vector of ones every time the left-hand side was

Â bigger than the right-hand side and a bunch of zeroes

Â every time the left-hand side was smaller than the right-hand side.

Â So, this vector now is just a vector of zeroes and ones,

Â one every time z plus square root n mu a minus mu 0 divided by sigma is

Â bigger then t divided by square root n minus 1 times square root xsq.

Â Okay, so that returns a vector of that type.

Â And then mean, the mean of a vector of zeros and ones is just the proportion of

Â ones, and you can type this in, the result is 60% or 60% up to Monte Carlo errors,

Â so you get 60 point something, of course.

Â by the way, the, the, you know, for these calculations, the error rate for the

Â mean is on the order of 1 over the square root of the sample size.

Â So, you know, we're getting a decimal place accuracy

Â on the order of 1 over square root 100,000.

Â so if you want higher accuracy, you gotta run the computer longer.

Â Okay.

Â So let's just go through these calculations.

Â I got my RStudio window open here. let's see.

Â So I'm going to first do the power t test, let me do that.

Â And here's the output right here.

Â it gives you, returns the, the values that you input, and you get power as 60%.

Â And then, to check, I'm going to do it here where,

Â [SOUND]

Â there you go. I, it was sd0 sigma.

Â So here, I set sd to be 4 and delta to

Â be 2, instead of delta to be 2 over

Â 4, and then it assumes that sd is 1.

Â and so here you can see, you get identically the same answer.

Â Here, let's see if I can, there you go. So, 60, 60.40329, yeah, same thing.

Â Okay now, let's go ahead and do our Monte Carlo

Â calculation. We'll define a number of simulations

Â and, and then here, let me just enter this in just to show you

Â [SOUND]

Â so I'm just going to show you a couple of values from this vector.

Â And see, it's not, it's actually not zeroes and ones.

Â It's trues and falses which are the Boolean values in R.

Â But, you know, in R, if, if, if, if you say, let's just multiply them by 1,

Â which does nothing it converts whatever it needs to convert them to, to numerical

Â values for operations, it just does it.

Â Okay, so let's take the mean of those values.

Â And it works out to be 60.608 which

Â is very close to the new very precise calculation.

Â Okay and then in these final, final slides, I just remind us that we

Â have to have a true mean and

Â a true standard deviation to do these calculations.

Â but that really, it's only the change in the

Â means divided by the standard deviations that affect the calculation.

Â So just reminding you of that point because it's incredibly useful.

Â All right, that's the end of Lecture 2, and we'll see you next time for Lecture 3.

Â Coursera provides universal access to the worldâ€™s best education,
partnering with top universities and organizations to offer courses online.