Analysis of Variance

Analysis of Variance
• In many research situations, we are interested in comparing the mean
scores of more than two groups
• In this case, we would use analysis of variance (ANOVA)
• One-way analysis of variance involves one IV (referred to as a factor),
which has different levels
• These levels correspond to different groups or conditions
• For example, if we are comparing the effectiveness of three different teaching
styles on students’ math scores, you would have one factor (teaching style)
with three levels (whole class activities, small group activities, self-paced
computer activities)
• The DV would be scores on a math test, which is continuous
Analysis of Variance
• Analysis of variance compares the variance (variability in scores)
between the different groups (believed to be due to the IV) with the
variability within each of the groups (believed to be due to chance)
• An F ratio is calculated, which represents the variance between the
groups divided by the variance within groups
• A large F ratio indicates that there is more variability between the groups
(caused by the IV) than there is within each group (referred to as the error
term)
• A significant F test indicates that we can reject the null hypothesis
• It does not, however, tell us which of the groups differ
Analysis of Variance
• To examine which groups differ, we need to conduct post-hoc tests
• Post-hoc tests are designed to help protect against the likelihood of
Type I Error
• Type I Error is where we reject the null hypothesis when we should retain it
• We are saying there is a significant difference, but there really is no significant difference
• Also called a false positive
• The probability of making a Type I Error is your p value, which is typically .05
• This means that you are willing to accept a 5% chance that you are wrong when you reject the
null hypothesis
• You reduce your risk of Type I Error when you lower your p value
• If p=.01, then you have a 1% chance that you are wrong when you reject the null
hypothesis
Type I & Type II Errors
• There are two possibilities when testing our hypothesis:
• 1. There is, in reality, an effect in our population OR
• 2. There is, in reality, no effect in the population
• We use test statistics and probability to tell us which of the two is most likely
• There are two mistakes we can make
• Type I Error: Occurs when we believe that there is a genuine effect in our population, when in fact there
isn’t
• If we use the conventional criterion then the probability of this error is .05 or 5%
• Assuming there is no effect in our population, if we replicated our data collection 100 times, we could expect that on five
occassions (or 5% of the time) we would obtain a test statistic large enough to make us think that there was a genuine
eeffect in the population even though there is no effect
• Type II Error: Occurs wheen we believe that there is no effect in the population when, in reality, there is
• There is a tradeoff between the two errors: If we lower the probability of accepting an effect
as genuine (make the p value smaller, like .01 or .001) then we increase the probability that
we will reject an effect that does genuinely exist
Type I Error & Multiple Tests
• Imagine you have 20 hypotheses and you want to use a significance level
of .05
• What is the probability of observing at least one significant result just
due to chance?
• P (at least one significant result) = 1 – P (no significant results)
• = 1 – (1-.05)20 = 0.64
• With 20 tests, we have a 64% chance of observing at least one significant result,
even if all of the tests are actually not significant
• Bonferroni correction sets the significance cut off at p/n = .05/20 = .0025
• You would use .0025 as your cut off point, not .05 for all 20 tests
One-Way Between Groups ANOVA with
Post-Hoc Tests
• One-way between groups ANOVA is used when you have one IV with
three or more levels and one continuous DV
• “One-way” indicates that there is only one IV
• “Between groups” means that you have different participants in each
of the groups
One-Way Between Groups ANOVA with
Post-Hoc Tests
• We will use survey.sav to explore the factors that affect respondents’
psychological adjustment, health and wellbeing
• Research Question: Is there a difference in optimism scores for young,
middle-aged and old participants?
• Requirements:
• One categorical IV with three or more distinct categories (Group 1= 18-29
years of age, Group 2 = 30-44 years of age, Group 3 = 45+ years of age =
Agegp3)
• One continuous DV (total optimism = toptim)
One-Way Between Groups ANOVA with
Post-Hoc Tests
• Analyze  Compare Means  One-way ANOVA
One-Way Between Groups ANOVA with
Post-Hoc Tests
• Move your dependent variable, toptim, to the Dependent List box
• Move your IV, agegp3, to the Factor box
One-Way Between Groups ANOVA with
Post-Hoc Tests
• Click on Options
• Select Descriptive, Homogeneity of
variance test, Brown-Forsythe, Welch,
and Means Plot
• For Missing values, make sure Exclude
cases analysis by analysis is selected
• Continue
One-Way Between Groups ANOVA with
Post-Hoc Tests
• Click on Post hoc and select Tukey
• The ANOVA will tell us whether there is significance
• The post hoc tests will tell us which groups are significantly different

• Continue  OK/Paste
One-Way Between Groups ANOVA with
Post-Hoc Tests
• Descriptives
• Provides information on each group
• Number of people in each group, means, standard deviation, minimum and maximum,
etc.
• Always check this table first
• Are the Ns for each group correct?
One-Way Between Groups ANOVA with
Post-Hoc Tests
• Test of Homogeneity of Variances
• The homogeneity of variance tests whether the variances in scores is the
same for each of the three groups
• Check the significance value for Levene’s test
• If this number is greater than .05, you have not violated the assumption of homogeneity
of variance
• In this example, the Sig. value is .475
One-Way Between Groups ANOVA with
Post-Hoc Tests
• Our data does not violate the homogeneity of variance assumption
• If you find that you have violated this assumption, you will need to
consult the table called Robust Tests for Equality of Mean
• The two tests here (Welsh and Brown-Forsythe) are preferable when the
assumption of the homogeneity of variance is violated
One-Way Between Groups ANOVA with
Post-Hoc Tests
• ANOVA
• The table gives both between-groups and within-groups sums of
squares, degrees of freedom, etc.
• The main thing you are interested in is the column called Sig.
• If the Sig. is less than to equal to .05, there is a significant difference somewhere
among the mean scores on your dependent variable for the three groups
• This does not tell you which group is different from which other group
• The statistical significance of the differences between each pair of
groups is provided in the Multiple Comparisons table
• Gives the results of the post-hoc tests
One-Way Between Groups ANOVA with
Post-Hoc Tests
• ANOVA
• In our example, the overall Sig. is .01, which is less than .05
• There is a statistically significant result somewhere among the groups
One-Way Between Groups ANOVA with
Post-Hoc Tests
• Multiple Comparisons
• You should look at this table only if you found a significant difference
in your overall ANOVA
• The post-hoc tests in this table will tell you exactly where the
differences among the groups occur
• Look at the Mean Difference column
• Look for any asterisks next to the values listed
• The asterisks indicate that the two groups being compared are significantly
different from one another at the p<.05 level
• The exact significance value is given in the Sig. column
One-Way Between Groups ANOVA with
Post-Hoc Tests
• Multiple Comparisons
• Our results indicates that:
• Groups 1 and 3 are statistically significantly difference from one another
• In other words, the 18-29 age group and the 45+ age group differ significantly
in terms of their optimism scores
One-Way Between Groups ANOVA with
Post-Hoc Tests
• Means Plots
• The plot provides an easy way to compare the mean scores for the
different groups
• You can see from this plot that the 18-29 age group recorded the
lowest optimism scores with the 45+ age group recording the highest
 One-Way Between Groups ANOVA with
Post-Hoc Tests
• Means Plots
• Warning: These plots can be misleading
• Depending on the the scale used on the Y axis,
even small differences can look dramatic
• In our example, the actual difference in the
mean scores between the groups is very small
• On the graph, the difference looks substantial
• Do not get too excited about your plots until
you have compared the mean values (in
Descriptives)
Effect Size
• Eta squared
• Eta squared = √(sum of squares between groups)/(total sum of squares)
• Eta squared = √179.07/8513.02 = √.02 =.14142 = .14

• Small = .10
• Medium = .30
• Large = .50

• .14 is a small effect size

One-Way Between Groups ANOVA with
Post-Hoc Tests
• Eta squared
• Warning: In this example, we obtained a statistically significant result,
but the actual difference in the mean scores of the groups was very
small (i.e., 21.36, 22.10, 22.96)
• This is evident in the small effect size obtained
• With a large enough sample (in this case, N=435) small differences can
become statistically significant, even if the difference between the groups is
of little practical importance
• Always interpret your results carefully, taking into account all the
information you have available
 Effect Size
• Eta squared is biased because it is based purely on sums of squares from the sample and no
adjustment is made for the fact that we are trying to estimate the effect size in the
population
• Instead, it is recommended we use omega squared ω2
• ω2 = sum of squares between groups – (dfsum of squares between groups) mean square within groups
---------------------------------------------------------------------------------------------------------------------
sum of squares total + mean square within groups

=179.07 – (2) 19.29 / 8513.02 + 19.29 =179.07 – 38.58 / 8532.31 = 140.49/8532.31 = .016465 = √.016 = .12649 = .13
• ω gives us a slightly lower estimate, which is also more accurate
• Interpretation for ω2
• Small = .01
• Medium = .06
• Large = .14
Presenting the Results in APA Style
• A one-way between-groups analysis of variance was conducted to explore the impact
of age on levels of optimism, as measured by the Life Orientation Test (LOT).
Participants were divided into three groups according to their age (Group 1: 29 years
of age or less, Group 2: 30 to 44 years of age, Group 3: 45 years of age and above).
• There was a statistically significant difference at the p<.05 level in LOT scores for the
three age groups, F(2,432) = 4.64, p=.01. Despite reaching statistical significance, the
actual difference in mean scores between the groups was quite small. The effect size
was small, ω2 =.016.
• Post-hoc comparisons using the Tukey HSD test indicated that the mean score for
Group 1 (M=21.36, SD=4.55) was significantly different from Group 3 (M=22.96,
SD=4.49), p=.007, 95% CI=[-2.83, -.36]. Group 2 (M=22.10, SD=4.15) did not differ
significantly from either Group 1 (p=.308, 95% CI=-.45,1.94 or Group 3 (p=.23, 95%
CI=-2.07,.37).
Breakdown of the Results
• There was a statistically significant difference at the p<.05 level in LOT
scores for the three age groups, F(2,432) = 4.64, p=.01.
Breakdown of the Results
• Despite reaching statistical significance, the actual difference in mean
scores between the groups was quite small. The effect size, calculated
using ω2 was .016
• =179.07 – (2) 19.29 / 8513.02 + 19.29 =179.07 – 38.58 / 8532.31 =
140.49/8532.31 = .016465 = .016
 Breakdown of the Results
• Post-hoc comparisons using the Tukey HSD test indicated that the mean
score for Group 1 (M=21.36, SD=4.55) was significantly different from
Group 3 (M=22.96, SD=4.49), p=.007, 95% CI=[-2.83, -.36]. Group 2
(M=22.10, SD=4.15) did not differ significantly from either Group 1 (p=.308,
95% CI=-.45,1.94 or Group 3 (p=.23, 95% CI=-2.07,.37).