Comparative Analyses of Pretest-Posttest Research Designs

Author(s): Donna R. Brogan and Michael H. Kutner

Source: The American Statistician, Vol. 34, No. 4 (Nov., 1980), pp. 229-232
Published by: Taylor & Francis, Ltd. on behalf of the American Statistical Association
Stable URL: http://www.jstor.org/stable/2684066
Accessed: 27-06-2016 08:43 UTC

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at
http://about.jstor.org/terms

JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content in a trusted
digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship. For more information about
JSTOR, please contact [email protected].

American Statistical Association, Taylor & Francis, Ltd. are collaborating with JSTOR to digitize,
preserve and extend access to The American Statistician

This content downloaded from 128.197.26.12 on Mon, 27 Jun 2016 08:43:29 UTC
All use subject to http://about.jstor.org/terms
In this section, The American Statistician publishes articles and section, articles and notes should be useful to a substantial number
notes of interest to teachers of the first mathematical statistics of teachers of such a course or should have the potential for
course and of applied statistics courses. To be suitable for this fundamentally affecting the way in which the course is taught.

Comparative Analyses of Pretest-Posttest Research Designs

DONNA R. BROGAN AND MICHAEL H. KUTNER*

Two common methods of analyzing data from a two-group pretest- a special case of the analysis of covariance where the
posttest research design are (a) two-sample t test on the difference regression coefficient of the posttest on the pretest is
score between pretest and posttest and (b) repeated-measures/
assumed to equal unity. Neter and Wasserman (1974,
split-plot analysis of variance. The repeated-measures/split-plot
analysis subsumes the t test analysis, although the former requires
p. 717) and Cox (1958, pp. 55-56) point out that if the
more assumptions to be satisfied. A numerical example is given common slope is not near one the covariance analysis
to illustrate some of the equivalences of the two methods of analysis. probably will be better than the difference score
The investigator should choose the method of analysis based on the analysis. We note that when an experimental group is
research objective(s).
to be compared to a control group, it is often likely
KEY WORDS: Repeated-measures/split-plot analysis; t test; Pre- that inequality of slopes will prevail among groups, thus
test-posttest designs. violating an assumption of the analysis of covariance.
Bock (1975, Sec. 7.3) compares the interpretation of a
1. INTRODUCTION difference-score analysis and covariance analysis and
suggests guidelines regarding which analysis to use.
A common research design is the two-group pretest/ Still another method of analyzing this design is to
view the pretest and posttest as a repeated-measures/
posttest design with one dependent variable where
subjects are not matched and may or may not be split-plot design or as a profile of two measurements

randomly assigned to the two groups (Cook and Camp- for each subject. Repeated-measures/split-plot designs
bell 1979). When the two groups are not formed by are discussed in detail by Winer (1971) and Steel and

random assignment of subjects, a random sample from Torrie (1980), whereas both repeated measures and

each of the two groups is necessary. This design can profile analysis are discussed in Morrison (1976, Secs.
be extended to more than two groups; an example is 4.5, 4.6, and 5.6).

the comparison of several different treatments with This article illustrates some of the equivalences and

each other or with a control group in which each group differences between the difference score analysis and

is measured on a pretest and posttest. the repeated-measures/split-plot or profile analysis.

The statistical analysis for these designs can be The numerical example and major discussion are for

approached from several viewpoints. If the dependent a two-group pretest/posttest design where subjects are
variable is measured on an interval or ratio scale, a not matched. Concluding remarks indicate how the re-

common analysis is to define a difference score for sults can be extended easily to more than two groups.

each subject (posttest minus pretest or vice versa) or

a relative difference measure (the difference divided
2. A NUMERICAL EXAMPLE
by the pretest) and then test the null hypothesis that
the means or medians of the (relative) differences are
We consider data from Rikkers et al. (1978), who
equal for each group. In many cases the t test or
report results of a prospective randomized surgical
analysis of variance is used, although nonparametric
trial allocating cirrhotic patients who had bled from
tests could also be used, for example, the Mann-
varices to either a nonselective shunt (standard opera-
Whitney U test, or the median test, or their analogs
tion) or to a selective shunt (new operation). The
for more than two groups.
dependent variable is the maximal rate of urea synthesis
Covariance analysis, where the pretest score is used
(MRUS), which is a quantitative test of liver function.
as the covariate, is another method used for analyzing
Poor liver function is associated with a low MRUS
this design. The difference score method is essentially
value. MRUS was measured preoperatively and early
postoperatively in eight selective shunt patients and
* Donna R. Brogan is Professor and Michael H. Kutner is Associ-
thirteen nonselective shunt patients. The purposes of
ate Professor in the Statistics and Biometry Department, School
the study were to assess preoperatively the compara-
of Medicine, Emory University, Atlanta, GA 30322. Work on this
article was partially supported by NCI Contract No. CB-74101 and bility of the selective and the nonselective groups and
USPHS Grant No. RR39. to longitudinally evaluate the change in liver function

C) The American Statistician, November 1980, Vol. 34, No. 4 229

This content downloaded from 128.197.26.12 on Mon, 27 Jun 2016 08:43:29 UTC
All use subject to http://about.jstor.org/terms
 1. Pre and Post Maximal Rate of Urea Synthesis /412 - 11 = 0
Level (mg urea N/hr/kg BW314) and Sample
/422- /21 = 0
Cell Means, by Group
and

Group Subject Pre Post /11 -21 = 0

Selective Shunt 1 51 48 Using an experiment wise error rate of .05, we con-

(new operation) 2 35 55 clude that the interaction effect is significant and
3 66 60
4 40 35
/21 is significantly greater than 922. Therefore, sig-
5 39 36 nificant deterioration of liver function occurred in the
6 46 43 nonselective patients between preoperative and early
7 52 46
postoperative evaluation periods, whereas the selective
8 42 54
group had no apparent deleterious effect. Two points
Mean All = 46.375 /L12 = 47.125 are worth noting in the example just cited: (a) The
equality of slopes test using the preoperative MRUS
Nonselective Shunt 9 34 16
(standard operation) 10 40 36 values as a covariate is rejected (p < .02); and (b) the
11 34 16 significant interaction effect requires special handling
12 36 18
when testing the last contrast since, for the pretest
13 38 32
14 32 14 level, we have a two-group experiment in which there
15 44 20 are no repeated measures. Therefore, the appropriate
16 50 43
error term for this type of comparison is MS (within
17 60 45
18 63 67 cell). For a more extensive coverage of this point the
19 50 36 reader is referred to Winer (1971, pp. 529-532).
20 42 34
21 43 32

Mean =21 43.538 422 = 31.462 3. REPEATED-MEASURES ANALYSIS

We now discuss the statistical properties of the

of the two groups. Table 1 reports the MRUS values repeated-measures analysis of variance for this
for each patient for the preoperative and postoperative example and compare it with the statistical properties
periods and the respective cell means. of the difference score analysis. Using the model pro-
For completeness Table 2 displays the standard posed by Winer (1971, p. 519), we have
repeated-measures analysis of variance table (analysis
Xijk = /U + ai + rk(i) + /3j + a/i j
of means method); in this example the total number of
subjects, n, is 21. The hypotheses of interest to the + 131jk(i) + Em(ijk) (3.1)
researchers were the interaction test and the simple
j = 1, 2 (pretest = 1, posttest = 2),
effects test on equality of preoperative population
means between groups. The test for interaction is i = 1, 2 (group 1 = 1, group 2 = 2),
significant (F = 11.36 with 1 and 19 df, p < .005);
k = 1, 2, . . . , n1, m = 1,
therefore, it is concluded that the pre/post average
change in the nonselective group is significantly differ- where Xijk is the observed value of subject k within
ent from the pre/post average change in the selective group i at time j,
group (see the figure). In the presence of a significant
, is the overall mean,
interaction effect it is generally of interest to test simple
ai is the effect of group i,
effects rather than main effects (Winer 1971, p. 529).
The Bonferroni multiple-comparison procedure (see
f1k(i) is the effect of subject k nested within group i,
,Bj is the effect of the repeated-measures variable] (i.e.,
Neter and Wasserman 1974) was adopted to test the
pretest and posttest),
following contrasts:
aj3ij is the interaction of group i with level j of the re-
G412 - 11) - (G22 - tk21) = 0 peated measures factor,

2. Repeated-Measures Analysis of Variance for Maximal Rate of Urea Synthesis Level

Mean
Source of Variation df Sum of Squares Squares F Ratio

Between Subjects 20 (n - 1)
Groups 1 847.48 847.48 (MSG) 3.63 (MSG/MSE)
Subjects Within Groups 19 (n - 2) 4440.00 233.68 (MSE)
Within Groups 21 (n)
Pre/Post 1 317.69 317.69 (MSP) 8.86 (MSP/MSPE)
Groups x Pre/Post 1 407.41 407.4 (MSGP) 11.36 (MSGP/MSPE)
(Pre/Post) x Subjects 19 (n - 2) 681.21 35.85 (MSPE)
Within Groups

230 C The American Statistician, November 1980, Vol. 34, No. 4

This content downloaded from 128.197.26.12 on Mon, 27 Jun 2016 08:43:29 UTC
All use subject to http://about.jstor.org/terms
and

13n1jk(i) is the interaction of subject k within group i with 50

level j of the repeated-measures factor.

The following constraints are imposed on the

parameters: coo
-t 0

a. = 3. a 8i. = aBJ3.j = 0,
where z40 "
af.j= a ,8ij, and so on. (3.2)

E N
In the design under discussion, the repeated-measures
factor and the group factor are each at two levels. - v e ~~~Selective Shunt
30 -
The general analysis of variance also is indicated in E ~~~o--o Nonselective Shuntb
Table 2, where n is the total number of subjects.
Note that it is not necessary for each group to contain
the same number of subjects. Assuming the group
factor and the pre/post factor to be fixed effects, Winer Pre (Before Su rgery) Post (After Su rgery)
(1971) shows that the appropriate F tests are as indi-
cated in the F ratio column of Table 2.
Mean Pre and Post />aximal Rate of Urea Syn-
It is worth noting exactly what null hypotheses are thesis Level (MRUS) by Type of Surgery
tested in Table 2. The ratio MSG/MSE tests the null
hypothesis that there is no group main effect. This is defined for each subject, the population variance of
equivalent to testing whether the sum of the pretest and the difference scores is the same for both groups. This
posttest observations on each subject has the same allows pooling over groups to calculate SSPE:.
population mean in the two groups. The ratio MSP/MSpE
tests the null hypothesis that there is no prelpost main
effect and is equivalent to testing whether the popula- 4. DIFFERENCE-SCORE ANALYSIS
tion mean of the pretest observations is the same as
the population mean of the posttest observations. The Using model (3.1) and forming a difference score
ratio MSGP/MSPE tests the null hypothesis that there is dik for each subject k nested in group i yields
no interaction between the group main effect and the
dik = Xilk- Xi2k
pre/post main effect. This ratio also tests whether the
difference between pretest and posttest observations = (,31 - P2) + (Befor Sureir) + Poslk(i) (t Srk(r)
has the same population mean in both groups. This is
+ (Em(ilk) EM(i2k)) * (4 .1 )
the test many researchers are interested in when using
this research design, since they often wish to assess The term (MR,US - a,8i2) iS a parameter associated with
whether a treatment has had any effect upon an experi- group i and measures the "effect" of group i on the
mental group. Note that this F test has (1, n - 2) df, difference score dik. The null hypothesis to be tested
which will correspond to the t test with (n - 2) df, as is How pooli - ove12 = g p21 - cal22 or Ho: l1 - a12
discussed in the next section. = Ui821 - md822 = a. The difference scores in (4.1) can
Two assumptions are required to arrive at the F tests be viewed as a one-way classification model in which
indicated in Table 2 (Winer 1971). the error term is the sum of the following two terms:

1. The pretest and posttest population variance- (WIlk(i) - P2k(i)) and (Em(ilk) - Em(2k))
covariance matrices for each group are assumed
If we assume this error term and also homogeneous
equal.
variances for the two groups, an appropriate test sta-
2. The random effects Hk(i), PIjk(i), and Em(Ujk) from
tistic is the Student t test for two independent samples
the model in (3.1) are all independently and
with (n - 2) df.
normally distributed with mean zero and variances
the erortrmi the sum of the followin two poteteifrms:
( cH2, (TO I2. and OE2, respectively.
5. COMPARISON OF THE TWO ANALYSES
Assumption (1), equality of the variance-covariance Ifo weassmchi sberro term an tw-amlso homogeneousedt
matrices, implies two other results worth noting. First, vracesmfore the togroups m ans appropriate thest sta-
The following three results are useful computationally
the variation of subjects within the two groups is titcaisltheStden t tes for0 two indtepsuaendent samples
and can be verified easily with the example from
homogeneous. That is, if each subject's pretest and F etfrGop nTbe2with (n1,12) df.
posttest observations are added together, this sum has
the same population variance in both groups. This
allows pooling over groups to calculate SSE. Second,
the variation of the interaction of subject and the pre!
post factor is homogeneous for the two groups. That is,
if the difference score between pretest and posttest is

C The American Statistician, November 1980, Vol. 34, No. 4 231

This content downloaded from 128.197.26.12 on Mon, 27 Jun 2016 08:43:29 UTC
All use subject to http://about.jstor.org/terms
 2. If the difference between the pretest and the post- of the main effect time (pre vs. post) if the Groups x
test is formed for each subject and a two-sample pre/post interaction had been nonsignificant. That is,
t test is used to compare the group means of these the nonsignificant interaction would have indicated that
differences, then the calculated t = 3.371 and is the two groups did not differ significantly on their
the square root of the F test for Groups x (pre/post) MRUS difference scores. The pre/post main effect test
in Table 2 with (1,19) df. would then indicate whether the MRUS difference
3. If all 21 subjects are considered to be in one group, score in both groups was significantly different from
then the t statistic to test the null hypothesis that zero, that is, whether the treatment did or did not
the mean difference score is zero has 20 df and effect both groups.
equals 3.158. From Table 2, if we reanalyze the
within-subjects component by assuming that the
Group x (pre/post) interaction is zero and SSGP is 6. GENERALIZATION OF FINDINGS
pooled with SSPE, then the F test for the main
effect pre/post yields F = 9.97 with (1,20) df, If there are more than two groups, similar results
which equals the square of the preceding t statistic. can be obtained. The difference-score analysis would
no longer be performed by a t test but by an F test in
These results demonstrate that the various F tests in
a one-way analysis of variance. A further extension
the repeated-measures analysis of variance can be
can be made where the different groups being compared
obtained by using simple t tests on linear combinations
may be defined by several factors in a factorial design.
of the pre and post scores. It can be shown algebraically
However, there is no logical extension of this dis-
that interpretations (1), (2), and (3) of the F tests hold
cussion to more than two levels of the repeated-
in the particular research design discussed in this
measures factor since a simple difference score analysis
article, that is, a two-group pretest/posttest design. In
would no longer be appropriate.
fact, the numerical operations in (1) and (2) of summing
and differencing the pretest/posttest observations are
[Received July 1977. Revised June 1980.]
used by the latest version of the BMD P2V program in
calculating sums of squares in repeated-measures de-
signs (Dixon and Brown 1979). REFERENCES
Since the difference-score analysis is embedded in
the repeated-measures analysis, the repeated-measures Bock, R. Darrell (1975), Multiv'ariate Statistical Methods in?
Behavioral Research, New York: McGraw-Hill Book Co.
analysis provides more information about the data at
Cook, Thomas D., and Campbell, Donald T. (1979), Quasi-
hand. Fewer assumptions, however, are required in the
Experimentation: Design and Analysis Issiues Jor Field Settings,
difference-score analysis. The difference-score analysis Chicago: Rand McNally.
assumes only homogeneous variances for the differ- Cox, D.R. (1958), Planning of Experiments, New York: John

ence scores and a normally distributed error term with Wiley & Sons.
Dixon, Wilfred J., and Brown, Morton B. (eds.) (1979), Biomedical
mean zero. It is easy to show that if the assumptions
Computer Programs P-Series, Berkeley: University of California
of the repeated-measures analysis are satisfied, then the Press.
assumptions of the difference-score analysis are also Morrison, Donald F. (1976), Multivariate Statistical Methods (2nd
met. However, the converse is not true. ed.), New York: McGraw-Hill Book Co.

In our experience, the researcher rarely is interested Neter, John, and Wasserman, William (1974), Applied Linear
Statistical Models, Homewood, Ill.: Richard D. Irwin.
in only the interaction test, that is, the difference-score
Rikkers, Layton F., Rudman, Daniel, Galambos, John T., Fulen-
analysis. Furthermore, simple effects are commonly of wider, J. Timothy, Milliken, William J., Kutner, Michael H.,
interest even in the no interaction effect experiments. Smith, Robert B., Salam, Atef A., Sones, Peter J., and Warren,
Therefore, we advocate the use of the repeated- W. Dean (1978), "A Randomized, Controlled Trial of the Distal

measures/split-plot analysis in most instances. How- Spenorenal Shunt," Annals of Surgery, 188, 271-282.
Steel, Robert G.D., and Torrie, James H. (1980), Principles anld
ever, we urge the user to empirically validate the
Procediures of Statistics (2nd ed.), New York: McGraw-Hill
underlying assumptions. Book Co.
In the example discussed in Section 2, the researchers Winer, B.J. (1971), Statistical Principles in Experimental Design
would have been interested in assessing the significance (2nd ed.), New York: McGraw-Hill Book Co.

232 (? The American Statistician, November 1980, Vol. 34, No. 4

This content downloaded from 128.197.26.12 on Mon, 27 Jun 2016 08:43:29 UTC
All use subject to http://about.jstor.org/terms