1. α and β and power, oh my! Type I and Type II errors Power and effect size Shown with animation

2. Significance Testing and the CLT Reject-or-not decision is based on CLT. CLT: based on “many samples” our sampling statistics will eventually make a normal (ish) distribution (assuming our conditions were met) We can describe the shape, the center, and the spread. And we can use what we know about normal curves to find the probability of getting any one particular sample statistic.

3. Significance Testing and the CLT For significance testing, we start—before we’ve even taken a sample—with a hypothesis about where the center is. Set a threshold where any thing past that, we’re officially “surprised”. That area is α, our significance level. α μ0 That threshold is our critical value. In the case of sample means, it’ll be t*. In the case of sample proportions, it’ll be z*. t*

4. Significance Testing and the CLT Take a sample. It is in keeping with our hypothesized distribution. It doesn’t mean our μ0 is right, but it certainly isn’t evidence against it. α μ0 FAIL TO REJECT H0. t*

5. Significance Testing and the CLT Take a sample. This statistic isn’t in keeping with our hypothesis. In fact, it’s a relatively rare occurrence. α μ0 t* Reject H0! Embrace Ha!

6. What if we made a mistake? Rare vs. never Normal distribution – still possible to get tails You can do everything right, and come to the mathematically correct answer and it is still a practically incorrect answer—the H0was right, you shouldn’t have rejected. α You have made a Type I error. How often will we make a Type I error? μ0 t* P(Type I) = α.

7. Rejected H0. What if we didn’t make a mistake? The true center is somewhere else. α Notice relationship between our statistic and the “true” distribution. μ0 μ X t*

8. What if we didn’t make a mistake? What is the probability of getting a sample statistic beyond our critical value threshold? P(Reject H0|H0 is wrong) = power of a test α μ0 μ The higher the power, the better the test is at rejecting the null hypothesis when it is wrong. t*

9. Can we still make a mistake, when our null hypothesis is wrong? What if we’d gotten a different sample? Look at “true” distribution: is this result possible? Rare? What would we do with H0? (remember, we don’t know the “true” distribution) FAIL TO REJECT H0 μ0 μ We have made a Type II error. t*

10. Can we still make a mistake, when our null hypothesis is wrong? What other kind of mistake can we make? P(Type II) = β β power μ0 μ t* Notice that βis the complement of the power. Power of the test = 1 – β.

11. Types of errors and their probabilities To recap: Type I error: the null hypothesis is correct, but we get a sample statistic that makes us reject H0. Probability: α Type II error: the null hypothesis is wrong (and the distribution is somewhere else), but we get a sample statistic that makes us fail to reject H0. power β α Probability: β μ0 μ Power: the probability of rejecting H0 when it is, in fact, wrong. t* Probability: 1 – β.

12. Types of errors and their probabilities How are they related? Power and β both rely on the “true” μ Calculating β is BTS  Effect size Depend on… power β α μ0 μ t*

13. Types of errors and their probabilities How does effect size relate to power and β? The larger the effect size… …the larger the power of the significance test …and the smaller the probability of making a Type II error (β). Effect size power β α μ0 μ Remember, we do not control the effect size—it is what it is. Just keep in mind its relation to both power and β. t*

15. telling a healthy man he’s sick

16. thinking an improvement works when it doesn’t

18. telling someone with an STD that they don’t have one

19. thinking a system improvement doesn’t work when it doesμ0 μ t*

20. Reducing the probabilities of errors If you decide that the cost of a Type I error is too great, how can you reduce the probability of making one? If you want to reduce the probability of a Type I error (false positive), merely reduce your significance level. β α μ0 μ That is, move your critical value threshold so that you have a lower α. t*

21. Reducing the probabilities of errors Reduce the probability of making a Type I error. If αdecreases, then β must increase. We still won’t calculate what βis, but we know it has increased. If βincreased, then power must decrease—since it’s 1 – β. β α μ0 μ t*

22. Reducing the probabilities of errors Reduce the probability of making a Type II error In order to decrease β, we must increase α Notice, also, that if βhas been reduced, then the power (1 – β) has increased. β α μ0 μ t*

23. Reducing the probabilities of errors So if you decrease the probability of Type I error, you increase the probability of a Type II error. And vice versa. Is there any way to reduce the probability of both? β α μ0 μ t* So what do we control?

24. Reducing the probabilities of errors We control the spread of our normal curves. CLT If our sample size increases, the centers don’t move, and we reduce variability… So by increasing n, our sample size, we’ve reduced bothαandβ. α β And if we’ve reduced β, we’ve increased the power (1 – β). μ0 μ t*

25. Review Vocabulary Type I error Type II error Power Effect size α β Critical value Significance level α β μ0 μ t*

26. Review Concepts How can you reduce the probability of a Type I error? What are the consequences? How can you reduce the probability of a Type II error? What are the consequences? How are effect size and power related? How are effect size and Type II errors related? How can you reduce the probability of both kinds of errors? α β μ0 μ t*