More on ANOVA

Learning Objectives

1. State what the Mean Square Error ($MSE$) estimates when the null hypothesis is true and when the null hypothesis is false
2. State what the Mean Square Between ($MSB$) estimates when the null hypothesis is true and when the null hypothesis is false
3. State the assumptions of a one-way ANOVA
4. Compute $MSE$
5. Compute $MSB$
6. Compute F and its two degrees of freedom parameters
7. Describe the shape of the F distribution
8. Explain why ANOVA is best thought of as a two-tailed test even though literally only one tail of the distribution is used
9. State the relationship between the t and F distributions
10. Partition the sums of squares into condition and error
11. Format data to be used with a computer statistics program

This section shows how ANOVA can be used to analyze a one-factor between-subjects design. We will use as our main example the "Smiles and Leniency" case study. In this study there were four conditions with 34 subjects in each condition. There was one score per subject. The null hypothesis tested by ANOVA is that the population means for all conditions are the same. This can be expressed as follows:

where H₀ is the null hypothesis and k is the number of conditions. In the "Smiles and Leniency" study, $k = 4$ and the null hypothesis is

If the null hypothesis is rejected, then it can be concluded that at least one of the population means is different from at least one other population mean.

Analysis of variance is a method for testing differences among means by analyzing variance. The test is based on two estimates of the population variance $(σ^2)$. One estimate is called the mean square error $MSE$ and is based on differences among scores within the groups. $MSE$ estimates $σ^2$ regardless of whether the null hypothesis is true (the population means are equal). The second estimate is called the mean square between $MSB$ and is based on differences among the sample means. $MSB$ only estimates $σ^2$ if the population means are equal. If the population means are not equal, then $MSB$ estimates a quantity larger than $σ^2$. Therefore, if the $MSB$ is much larger than the $MSE$, then the population means are unlikely to be equal. On the other hand, if the $MSB$ is about the same as $MSE$, then the data are consistent with the null hypothesis that the population means are equal.

Before proceeding with the calculation of $MSE$ and $MSB$, it is important to consider the assumptions made by ANOVA:

1. The populations have the same variance. This assumption is called the assumption of homogeneity of variance.
2. The populations are normally distributed.
3. Each value is sampled independently from each other value. This assumption requires that each subject provide only one value. If a subject provides two scores, then the values are not independent. The analysis of data with two scores per subject is shown in the section on within-subjects ANOVA later in this chapter.

These assumptions are the same as for a $t$ test of differences between groups except that they apply to two or more groups, not just to two groups.

The means and variances of the four groups in the "Smiles and Leniency" case study are shown in Table 1. Note that there are 34 subjects in each of the four conditions (False, Felt, Miserable, and Neutral).

Sample Sizes

Computing $MSE$

Recall that the assumption of homogeneity of variance states that the variance within each of the populations ($σ^2$) is the same. This variance, $σ^2$, is the quantity estimated by $MSE$ and is computed as the mean of the sample variances. For these data, the $MSE$ is equal to 2.6489.

Computing $MSB$

The formula for $MSB$ is based on the fact that the variance of the sampling distribution of the mean is

where $n$ is the sample size of each group. Rearranging this formula, we have

Therefore, if we knew the variance of the sampling distribution of the mean, we could compute $σ^2$ by multiplying it by $n$. Although we do not know the variance of the sampling distribution of the mean, we can estimate it with the variance of the sample means. For the leniency data, the variance of the four sample means is 0.270. To estimate $σ^2$, we multiply the variance of the sample means (0.270) by n (the number of observations in each group, which is 34). We find that $MSB$ = 9.179.

To sum up these steps:

1. Compute the means.
2. Compute the variance of the means.
3. Multiply the variance of the means by $n$.
   
   

Recap

If the population means are equal, then both $MSE$ and $MSB$ are estimates of $σ^2$ and should therefore be about the same. Naturally, they will not be exactly the same since they are just estimates and are based on different aspects of the data: The $MSB$ is computed from the sample means and the $MSE$ is computed from the sample variances.

If the population means are not equal, then $MSE$ will still estimate $σ^2$  because differences in population means do not affect variances. However, differences in population means affect $MSB$ since differences among population means are associated with differences among sample means. It follows that the larger the differences among sample means, the larger the $MSB$. In short, $MSE$ estimates $σ^2$ whether or not the population means are equal, whereas $MSB$ estimates $σ^2$ only when the population means are equal and estimates a larger quantity when they are not equal.

Comparing $MSE$ and $MSB$

The critical step in an ANOVA is comparing $MSE$ and $MSB$. Since $MSB$ estimates a larger quantity than $MSE$ only when the population means are not equal, a finding of a larger $MSB$ than an $MSE$ is a sign that the population means are not equal. But since $MSB$ could be larger than $MSE$ by chance even if the population means are equal, $MSB$ must be much larger than $MSE$ in order to justify the conclusion that the population means differ. But how much larger must $MSB$ be? For the "Smiles and Leniency" data, the $MSB$ and $MSE$ are 9.179 and 2.649, respectively. Is that difference big enough? To answer, we would need to know the probability of getting that big a difference or a bigger difference if the population means were all equal. The mathematics necessary to answer this question were worked out by the statistician R. Fisher. Although Fisher's original formulation took a slightly different form, the standard method for determining the probability is based on the ratio of $MSB$ to $MSE$. This ratio is named after Fisher and is called the $F$ ratio.

For these data, the $F$ ratio is

Therefore, the $MSB$ is 3.465 times higher than $MSE$. Would this have been likely to happen if all the population means were equal? That depends on the sample size. With a small sample size, it would not be too surprising because results from small samples are unstable. However, with a very large sample, the $MSB$ and $MSE$ are almost always about the same, and an F ratio of 3.465 or larger would be very unusual. Figure 1 shows the sampling distribution of F for the sample size in the "Smiles and Leniency" study. As you can see, it has a positive skew.

Figure 1. Distribution of $F$.

From Figure 1, you can see that F ratios of 3.465 or above are unusual occurrences. The area to the right of 3.465 represents the probability of an F that large or larger and is equal to 0.018. In other words, given the null hypothesis that all the population means are equal, the probability value is 0.018 and therefore the null hypothesis can be rejected. The conclusion that at least one of the population means is different from at least one of the others is justified.

The shape of the F distribution depends on the sample size. More precisely, it depends on two degrees of freedom ($df$) parameters: one for the numerator ($MSB$) and one for the denominator ($MSE$). Recall that the degrees of freedom for an estimate of variance is equal to the number of observations minus one. Since the $MSB$ is the variance of $k$ means, it has $k - 1 df$. The $MSE$ is an average of $k$ variances, each with $n - 1 df$. Therefore, the $df$ for $MSE$ is $k(n - 1) = N - k$, where $N$ is the total number of observations, $n$ is the number of observations in each group, and k is the number of groups. To summarize:

For the "Smiles and Leniency" data,

The $F$ distribution calculator shows that $p = 0.018$.

One-Tailed or Two?

Is the probability value from an $F$ ratio a one-tailed or a two-tailed probability? In the literal sense, it is a one-tailed probability since, as you can see in Figure 1, the probability is the area in the right-hand tail of the distribution. However, the $F$ ratio is sensitive to any pattern of differences among means. It is, therefore, a test of a two-tailed hypothesis and is best considered a two-tailed test.

Relationship to the t test

Since an ANOVA and an independent-groups t test can both test the difference between two means, you might be wondering which one to use. Fortunately, it does not matter since the results will always be the same. When there are only two groups, the following relationship between F and t will always hold:

where $dfd$ is the degrees of freedom for the denominator of the $F$ test and $df$ is the degrees of freedom for the $t$ test. $dfd$ will always equal $df$.

Sources of Variation

Why do scores in an experiment differ from one another? Consider the scores of two subjects in the "Smiles and Leniency" study: one from the "False Smile" condition and one from the "Felt Smile" condition. An obvious possible reason that the scores could differ is that the subjects were treated differently (they were in different conditions and saw different stimuli). A second reason is that the two subjects may have differed with regard to their tendency to judge people leniently. A third is that, perhaps, one of the subjects was in a bad mood after receiving a low grade on a test. You can imagine that there are innumerable other reasons why the scores of the two subjects could differ. All of these reasons except the first (subjects were treated differently) are possibilities that were not under experimental investigation and, therefore, all of the differences (variation) due to these possibilities are unexplained. It is traditional to call unexplained variance error even though there is no implication that an error was made. Therefore, the variation in this experiment can be thought of as being either variation due to the condition the subject was in or due to error (the sum total of all reasons the subjects' scores could differ that were not measured).

One of the important characteristics of ANOVA is that it partitions the variation into its various sources. In ANOVA, the term sum of squares (SSQ) is used to indicate variation. The total variation is defined as the sum of squared differences between each score and the mean of all subjects. The mean of all subjects is called the grand mean and is designated as GM. (When there is an equal number of subjects in each condition, the grand mean is the mean of the condition means.) The total sum of squares is defined as

which means to take each score, subtract the grand mean from it, square the difference, and then sum up these squared values. For the "Smiles and Leniency" study, $\mathrm{SSQ}_{\text {total}}=\sum(X-G M)^{2} = 377.19$.

The sum of squares condition is calculated as shown below.

where n is the number of scores in each group, $k$  is the number of groups, $M_1$ is the mean for Condition 1, $M_2$ is the mean for Condition 2, and $M_k$ is the mean for Condition k. For the Smiles and Leniency study, the values are:

If there are unequal sample sizes, the only change is that the following formula is used for the sum of squares condition:

where $n_i$ is the sample size of the ith condition. $SSQ_{total}$ is computed the same way as shown above.

The sum of squares error is the sum of the squared deviations of each score from its group mean. This can be written as

where $X_{i1}$ is the ith score in group 1 and $M_1$ is the mean for group 1, $X_{i2}$ is the ith score in group 2 and $M_2$ is the mean for group 2, etc. For the "Smiles and Leniency" study, the means are: 5.368, 4.912, 4.912, and 4.118. The SSQerror is therefore:

The sum of squares error can also be computed by subtraction:

Therefore, the total sum of squares of 377.19 can be partitioned into $SSQ_{condition}$ (27.53) and $SSQ_{error}$ (349.66).

Once the sums of squares have been computed, the mean squares ($MSB$ and $MSE$) can be computed easily. The formulas are:

where dfn is the degrees of freedom numerator and is equal to $k - 1 = 3$.

which is the same value of $MSB$ obtained previously (except for rounding error). Similarly,

where dfd is the degrees of freedom for the denominator and is equal to $N - k$.

which is the same as obtained previously (except for rounding error). Note that the dfd is often called the dfe for degrees of freedom error.

The Analysis of Variance Summary Table shown below is a convenient way to summarize the partitioning of the variance. The rounding errors have been corrected.

The first column shows the sources of variation, the second column shows the degrees of freedom, the third shows the sums of squares, the fourth shows the mean squares, the fifth shows the $F$ ratio, and the last shows the probability value. Note that the mean squares are always the sums of squares divided by degrees of freedom. The $F$ and $p$ are relevant only to Condition. Although the mean square total could be computed by dividing the sum of squares by the degrees of freedom, it is generally not of much interest and is omitted here.

Formatting Data for Computer Analysis

Most computer programs that compute ANOVAs require your data to be in a specific form. Consider the data in Table 3.

             Df Sum Sq Mean Sq F value Pr(>F)
leniency.f    3   27.5   9.178   3.465 0.0182
Residuals   132  349.7   2.649

Learning Objectives

1. Define a within-subjects factor
2. Explain why a within-subjects design can be expected to have more power than a between-subjects design
3. Be able to create the Source and df columns of an ANOVA summary table for a one-way within-subjects design
4. Explain error in terms of interaction
5. Discuss the problem of carryover effects
6. Be able to create the Source and df columns of an ANOVA summary table for a design with one between-subjects and one within-subjects variable
7. Define sphericity
8. Describe the consequences of violating the assumption of sphericity
9. Discuss courses of action that can be taken if sphericity is violated

Within-subjects factors involve comparisons of the same subjects under different conditions. For example, in the "ADHD Treatment" study, each child's performance was measured four times, once after being on each of four drug doses for a week. Therefore, each subject's performance was measured at each of the four levels of the factor "Dose". Note the difference from between-subjects factors for which each subject's performance is measured only once and the comparisons are among different groups of subjects. A within-subjects factor is sometimes referred to as a repeated-measures factor since repeated measurements are taken on each subject. An experimental design in which the independent variable is a within-subjects factor is called a within-subjects design.

Advantage of Within-Subjects Designs

One-Factor Designs

Let's consider how to analyze the data from the "ADHD Treatment" case study. These data consist of the scores of 24 children with ADHD on a delay of gratification (DOG) task. Each child was tested under four dosage levels. For now, we will be concerned only with testing the difference between the mean in the placebo condition (the lowest dosage, D0) and the mean in the highest dosage condition (D60). The details of the computations are relatively unimportant since they are almost universally done by computers. Therefore we jump right to the ANOVA Summary table shown in Table 1.

The first source of variation, "Subjects," refers to the differences among subjects. If all the subjects had exactly the same mean (across the two dosages), then the sum of squares for subjects would be zero; the more subjects differ from each other, the larger the sum of squares subjects.

Dosage refers to the differences between the two dosage levels. If the means for the two dosage levels were equal, the sum of squares would be zero. The larger the difference between means, the larger the sum of squares.

The error reflects the degree to which the effect of dosage is different for different subjects. If subjects all responded very similarly to the drug, then the error would be very low. For example, if all subjects performed moderately better with the high dose than they did with the placebo, then the error would be low. On the other hand, if some subjects did better with the placebo while others did better with the high dose, then the error would be high. It should make intuitive sense that the less consistent the effect of dosage, the larger the dosage effect would have to be in order to be significant. The degree to which the effect of dosage differs depending on the subject is the Subjects $x$ Dosage interaction. Recall that an interaction occurs when the effect of one variable differs depending on the level of another variable. In this case, the size of the error term is the extent to which the effect of the variable "Dosage" differs depending on the level of the variable "Subjects". Note that each subject is a different level of the variable "Subjects".

Other portions of the summary table have the same meaning as in between-subjects ANOVA. The F for dosage is the mean square for dosage divided by the mean square error. For these data, the $F$ is significant with $p = 0.004$. Notice that this F test is equivalent to the t test for correlated pairs, with $F = t^2$.

Table 2 shows the ANOVA Summary Table when all four doses are included in the analysis. Since there are now four dosage levels rather than two, the df for dosage is three rather than one. Since the error is the Subjects $x$ Dosage interaction, the df for error is the df for "Subjects" (23) times the df for Dosage (3) and is equal to 69.

Carryover Effects

Often performing in one condition affects performance in a subsequent condition in such a way as to make a within-subjects design impractical. For example, consider an experiment with two conditions. In both conditions subjects are presented with pairs of words. In Condition A, subjects are asked to judge whether the words have similar meaning whereas in Condition B, subjects are asked to judge whether they sound similar. In both conditions, subjects are given a surprise memory test at the end of the presentation. If Condition were a within-subjects variable, then there would be no surprise after the second presentation and it is likely that the subjects would have been trying to memorize the words.

Not all carryover effects cause such serious problems. For example, if subjects get fatigued by performing a task, then they would be expected to do worse on the second condition they were in. However, as long as the order of presentation is counterbalanced so that half of the subjects are in Condition A first and Condition B second, the fatigue effect itself would not invalidate the results, although it would add noise and reduce power. The carryover effect is symmetric in that having Condition A first affects performance in Condition B to the same degree that having Condition B first affects performance in Condition A.

Asymmetric carryover effects cause more serious problems. For example, suppose performance in Condition B were much better if preceded by Condition A, whereas performance in Condition A was approximately the same regardless of whether it was preceded by Condition B. With this kind of carryover effect, it is probably better to use a between-subjects design.

One between- and one within-subjects factor

In the "Stroop Interference" case study, subjects performed three tasks: naming colors, reading color words, and naming the ink color of color words. Some of the subjects were males and some were females. Therefore, this design had two factors: gender and task. The ANOVA Summary Table for this design is shown in Table 3.

The computations for the sums of squares will not be covered since computations are normally done by software. However, there are some important things to learn from the summary table. First, notice that there are two error terms: one for the between-subjects variable Gender and one for both the within-subjects variable Task and the interaction of the between-subjects variable and the within-subjects variable. Typically, the mean square error for the between-subjects variable will be higher than the other mean square error. In this example, the mean square error for Gender is about twice as large as the other mean square error.

The degrees of freedom for the between-subjects variable is equal to the number of levels of the between-subjects variable minus one. In this example, it is one since there are two levels of gender. Similarly, the degrees of freedom for the within-subjects variable is equal to the number of levels of the variable minus one. In this example, it is two since there are three tasks. The degrees of freedom for the interaction is the product of the degrees of freedom for the two variables. For the Gender x Task interaction, the degrees of freedom is the product of degrees of freedom Gender (which is 1) and the degrees of freedom Task (which is 2) and is equal to 2.

Assumption of Sphericity

Within-subjects ANOVA makes a restrictive assumption about the variances and the correlations among the dependent variables. Although the details of the assumption are beyond the scope of this book, it is approximately correct to say that it is assumed that all the correlations are equal and all the variances are equal. Table 4 shows the correlations among the three dependent variables in the "Stroop Interference" case study.

Note that the correlation between the word reading and the color naming variables of 0.7013 is much higher than the correlation between either of these variables with the interference variable. Moreover, as shown in Table 5, the variances among the variables differ greatly.

Naturally the assumption of sphericity, like all assumptions, refers to populations not samples. However, it is clear from these sample data that the assumption is not met in the population.

Consequences of Violating the Assumption of Sphericity

Although ANOVA is robust to most violations of its assumptions, the assumption of sphericity is an exception: Violating the assumption of sphericity leads to a substantial increase in the Type I error rate. Moreover, this assumption is rarely met in practice. Although violations of this assumption had at one time received little attention, the current consensus of data analysts is that it is no longer considered acceptable to ignore them.

Approaches to Dealing with Violations of Sphericity

If an effect is highly significant, there is a conservative test that can be used to protect against an inflated Type I error rate. This test consists of adjusting the degrees of freedom for all within-subjects variables as follows: The degrees of freedom numerator and denominator are divided by the number of scores per subject minus one. Consider the effect of Task shown in Table 3. There are three scores per subject and therefore the degrees of freedom should be divided by two. The adjusted degrees of freedom are:

The probability value is obtained using the F probability calculator with the new degrees of freedom parameters. The probability of an F of 228.06 or larger with 1 and 45 degrees of freedom is less than 0.001. Therefore, there is no need to worry about the assumption violation in this case.

Possible violation of sphericity does make a difference in the interpretation of the analysis shown in Table 2. The probability value of an F of 5.18 with 1 and 23 degrees of freedom is 0.032, a value that would lead to a more cautious conclusion than the p value of 0.003 shown in Table 2.

The correction described above is very conservative and should only be used when, as in Table 3, the probability value is very low. A better correction, but one that is very complicated to calculate, is to multiply the degrees of freedom by a quantity called ε (the Greek letter epsilon). There are two methods of calculating ε. The correction called the Huynh-Feldt (or H-F) is slightly preferred to the one called the Greenhouse-Geisser (or G-G), although both work well. The G-G correction is generally considered a little too conservative.

A final method for dealing with violations of sphericity is to use a multivariate approach to within-subjects variables. This method has much to recommend it, but it is beyond the scope of this text.

Learning Objectives

1. Define main effect, simple effect, interaction, and marginal mean
2. State the relationship between simple effects and interaction
3. Compute the source of variation and $df$ for each effect in a factorial design
4. Plot the means for an interaction
5. Define three-way interaction
   
   

Basic Concepts and Terms

1. Weight: Are applicants judged differently depending on the weight of their companion?
2. Relationship: Are applicants judged differently depending on their relationship with their companion?
3. Weight x Relationship Interaction: Does the effect of weight differ depending on the relationship with the companion?

		Companion Weight
		Obese	Typical	Marginal Mean
Relationship	Girlfriend	5.65	6.19	5.92
	Acquaintance	6.15	6.59	6.37
	Marginal Mean	5.90	6.39

Tests of Significance

Source	df	SSQ	MS	F	p
Weight	1	10.4673	10.4673	6.214	0.0136
Relation	1	8.8144	8.8144	5.233	0.0234
W x R	1	0.1038	0.1038	0.062	0.8043
Error	172	289.7132	1.6844
Total	175	310.1818

		Companion Weight
		Obese	Typical
Relationship	Girlfriend	40	42
	Acquaintance	40	54

Plotting Means

An Example with Interaction

		Esteem
		High	Low
Outcome	Success	7	6
		8	5
		7	7
		8	4
		9	5
		5	6
	Failure	4	9
		6	8
		5	9
		4	8
		7	7
		3	6

Source	df	SSQ	MS	F	p
Outcome	1	0.0417	0.0417	0.0256	0.8744
Esteem	1	2.0417	2.0417	1.2564	0.2756
O x E	1	35.0417	35.0417	21.5641	0.0002
Error	20	32.5000	1.6250
Total	23	69.6250

Three-Factor Designs

Source	df	SSQ	MS	F	p
Hole	1	8258.27	8258.27	266.68	<0.0001
Assembly	1	13369.14	13369.14	431.73	<0.0001
H x A	1	2848.89	2848.89	92.00	<0.0001
Barrel	1	35.0417	35.0417	21.5641	<0.0001
H x B	1	594.14	594.14	19.1865	<0.0001
A x B	1	135.14	135.14	4.36	0.0413
H x A x B	1	1396.89	1396.89	45.11	<0.0001
Error	56	1734.12	30.97
Total	63	221386.91

Formatting Data for Computer Analysis

outcome	esteem	attrib
1	1	7
1	1	8
1	1	7
1	1	8
1	1	9
1	1	5
1	2	6
1	2	5
1	2	7
1	2	4
1	2	5
1	2	6
2	1	4
2	1	6
2	1	5
2	1	4
2	1	7
2	1	3
2	2	9
2	2	8
2	2	9
2	2	8
2	2	7
2	2	6

1. Click the "Enter/Edit Data" button. (You may be warned that for security reasons you must use the keyboard shortcut for pasting data).
2. Paste your data.
3. Click "Accept Data".
4. Click the "Advanced" button next to the "ANOVA" button.
5. Select "attrib" as the dependent variable and both "outcome" and "esteem" as "group" variables.
6. Click the "Do ANOVA" button.

Learning Objectives

1. State why unequal $n$ can be a problem
2. Define confounding
3. Compute weighted and unweighted means
4. Distinguish between Type I and Type III sums of squares
5. Describe why the cause of the unequal sample sizes makes a difference in the interpretation

The Problem of Confounding

Whether by design, accident, or necessity, the number of subjects in each of the conditions in an experiment may not be equal. For example, the sample sizes for the "Bias Against Associates of the Obese" case study are shown in Table 1. Although the sample sizes were approximately equal, the "Acquaintance Typical" condition had the most subjects. Since $n$ is used to refer to the sample size of an individual group, designs with unequal sample sizes are sometimes referred to as designs with unequal $n$.

Table 1. Sample Sizes for "Bias Against Associates of the Obese" Study.

Weighted and Unweighted Means

Types of Sums of Squares

Which Type of Sums of Squares to Use (optional)

Unweighted Means Analysis

Causes of Unequal Sample Sizes

Learning Objectives

1. Compute Tukey HSD test
2. Describe an interaction in words
3. Describe why one $M_i$ ght want to compute simple effect tests following $A$ significant interaction

Main Effects

Interactions

Describing Interactions

Simple Effect Tests

Components of Interaction (optional)