The Alpha Level and Statistical Power Demystified

pha Level and Statistical Power Demystified Brent L. Arnold, PhD, ATC Cuwy School of Education, University of Virginia f you have ever plowed through the methods section of a research manuscript, you probably came across a reference to the alpha level (usually alpha = .05) of the statistics used. Or perhaps you noticed in the results section that a statistic was reported as being significant at the p < .05 level. What do these numbers mean, and how do they help the reader? To understand this concept, you must first realize that statisticians can only tell you with certainty what will happen when nothing is happening. Our entire statistical system is based on probabilities that occur when nothing happens (i.e., the null hypothesis). Let me give you an example. If I randomly pick a group of 50 people from a population with ankle sprains and measure their ankle swelling in milliliters of volume, calculate the group's average swelling rate, put them back in the population, randomly pick again, and repeat this process hundreds of times, I end up with hundreds of averages for swelling. Now, if I randomly pick two of these swelling averages, or means, calculate the difference between them, put them back "in the hat," and repeat this process hundreds of times, I eventually have hundreds of difference scores. David H. Perrin, PhD, ATC, Column Editor So what? Well, interestingly enough, these difference scores begin to fall into a pattern. Specifically, small differences occur most often. In fact small differences occur 68% of the time, medium differences occur 27% of the time, and large differences occur only 5% of the time, approximately. (In statistics, everything has qualifiers.) Going back to the ankle sprains, I know that by pure chance, 5% of the time I will get large differences in the amounts of swelling between randomly selected groups. As a researcher, it is this last 5% (or alpha = .05, or p < .05) in which I am interested. If as an athletic therapist I want to know whether ice actually reduces swelling, I might randomly select people with ankle sprains to be either treated with ice (treatment group) or not treated with ice (control group). At the end of the study I can calculate the difference in swelling between both groups and compare this value to what I know I will get - when the means are created randomly, as they were above. If the difference between my control and treatment groups is large, then I know that the difference occurs only 5% of the time by chance. Well, quite frankly, I'm just not that lucky. Therefore, if I ended up with a difference that was large, it must have been due to the treatment and not to chance. Thus I conclude that my ice treatment worked. In other words, the alpha level or probability (p) level of a study tells the reader how likely the researcher would have gotten the same result by chance. If the level is .05, then there is a 5% chance the results occurred by chance; if the level is set at .01, another frequently used level, then there is only a 1% chance that the results occurred by chance. Based on this, it should be obvious that the smaller the alpha level, the more certain the researcher is that the result is not due to chance. Thus, you might wonder, why not make the alpha as small as possible to increase certainty? Well, one of the compromises made by increasing the alpha level is decreasing statistical power, that is, decreasing the chances of finding a statistical difference. Statistical power can be likened to a magnifying glass. As you increase the power, it becomes easier to see more details o r smaller differences. Conversely, if you are looking for large differences, a magnlfylng glass may not be needed. Statistics work the same way. The less power they 43 1996 Human Kinetics 36 Athletic Therapy Today July 1996 Table I I Four Ways to Create Statistical Power -- I YOU CAN'T STOP A BLITZING LINEBACKER AT RAMMING SPEED. Method (BUT YOU CAN SOFTEN THE - Increase the size of the alpha Downloaded by York Univ Libraries on 09/17/16, Volume 1, Article Number 4 - Increase the size of the difference between the groups being studied, by increasing the potency of the treatment - Increase the consistency of the treatment application or measurement procedure, thus decreasing the variability within the 2 groups - Increase the number of people participating in this study have, the less they can detect. In fact it is possible to have so little power that a research study has less than a 50% chance of finding statistically significant differences even when there are large clinical differences. Thus the researcher usually tries to maximize statistical power. Statistical power can be created in one of four ways, as shown in Table 1.Just as it is possible to have too little power, it is also possible to have so much power that small, meaningless differences are detected. For example, if in the ankle swelling study I enlisted 300 subjects to participate, it might be possible to detect differences as small as a few milliliters of volume. From a clinical perspective, it is difficult to believe that a few milliliters is important. Nevertheless, with enough people, it might be statisticallysignificant. Thus the researcher is always trying to balance the desire to be certain of the results with the need to keep the statistical power from being reJuly 1996 BLOW.) -Decreases the certaintyof the results - Sometimes the potency cannot be increased without increasing the detrimental effects - Usually cannot be determined until after the study, if it can be determined at all - This is the easiest way to increase power and is the one most often used duced to the point of not revealing meaningful differences. At the same time, he or she must be careful not to create so much power as to detect very small differences. In this way statistics is more of an art than a science. Finally, I want to emphasize the difference between clinical significance and statistical significance. The latter only indicates that the results of the research study are not likely based on chance as determined by the alpha level. It does not indicate that the results are meaningful or important. Thus it is still up to you, as the reader, to judge whether the results are clinically important. In doing so, you must keep in mind that a researcher who creates too much power may find small, meaningless differences. Brent L. Arnold is an assistant professor in the Curry School of Education at the University of Virginia. He obtained a PhD in sports medicine from the University of Virginia and has worked as a staff athletic trainer at Princeton. Athletic Therapy Today When your player is suffering from a hip pointer or AC contusion, every hit hurts. But now, you can offer protection with amazing new padding from Cramer. O Ortho GelTMis a visco-elastic polymer that adheres to skin. When a linebacker strikes, Ortho Gel spreads out carrying the impact away from the injured area. When the force subsides, Ortho Gel returns to its original shape - ready to cushion another blow. O Unlike foam, our new gel won't bottom out. Plus, it can even provide cold l(Fpptherapy.* Just pop it in W" the freezer, then cut a pad to fit. CI Put Ortho Gel on your team today. It bounces back, so your athletes can, too. @ 1-800-345-2231 'please observe the necessary precaut~ons to prevent thermal Injury Q 1996, Cramer Products. Inc ' Ortho Gel 1s a registered trademark of Cramer Products, Inc 37