15 - 1

Chapter 15. Nonexperimental Research Designs: Correlational Design, Ex Post

Facto Design, Naturalistic Observation, and Qualitative Research

Introduction to Nonexperimental Designs

Correlational Design
Importance of Correlational Research
Direction of Control and Third Variable Problems
Addressing Directionality and Third Variable Problems
Correlational Ruling Out Factors
Interpretation of Correlational Data

Ex Post Facto Design

Naturalistic Observation
Qualitative Research
Case Study
Phenomenology
Ethnography

Case Analysis
General Summary
Detailed Summary
Key Terms
Review Questions/Exercises
 15 - 2

Introduction to Nonexperimental Designs

We have said much about true experiments and we have described their strength in drawing strong,
confident conclusions. A word of caution is advisable. An experiment may use random assignment and
involve manipulation of the treatment variable and still be essentially worthless as a basis for drawing
conclusions. It is essential that rigorous controls, careful execution, planning, thoughtfulness, etc.,
accompany a valid design. We have also noted the qualities of designs termed quasi-experimental. Recall
that these were characterized as designs in which the independent variable was manipulated but the study
lacked random assignment of participants to conditions.
As we have seen thus far in the book, experimental research is a very powerful tool for generating a
scientific database for drawing cause-effect conclusions, for testing hypotheses and evaluating theory, for
answering questions and satisfying our intellectual curiosity, for systematic manipulation of variables, and,
at times, for discovering principles that may be relevant to everyday life. After considerable discussion of
the virtues of experimental designs, you might wonder why researchers would use other types of
nonexperimental designs. Actually, there are several good reasons to use nonexperimental designs. Many
very interesting questions in psychology do not lend themselves to experimental designs. Some of these
questions involve independent variables that simply cannot be manipulated by a researcher. If we wish to
study the effects on a dependent measure of such naturally occurring variables as gender, ethnic
background, intelligence, temperament, or body size, we cannot say to the participants, "For the purposes
of this experiment, I am going to declare you a female, or a black, or a person with an IQ of 130!” In
addition, some questions involve independent variables that could theoretically be manipulated by a
researcher but are not because the opportunity does not present itself, the financial cost would be too high,
or the ethical concerns too great. For example, we may ask, "Do individuals who have left hemispheric
brain damage show greater verbal impairment than those who have comparable damage to the right
hemisphere?" Obviously, it is not possible to randomly assign people to an experimental and control group
and then conduct brain surgery to answer this question. However, if we are to shed any light on the
question, we are forced to look into the histories of people who have suffered brain damage as a result of
adverse circumstances. Similarly, as we have repeatedly explored the issue of TV violence and aggressive
behavior in children, we would certainly be interested in the effects of long-term (in terms of years)
exposure to TV violence. I’m sure that you can see the ethical issues involved in randomly assigning a
group of children to watch violent television for several years!
Thus, although nonexperimental research designs are not as powerful as experimental designs i.e., do
not rule out as many alternative hypothesis (explanations), they provide us with options for pursuing
 15 - 3

interesting and important questions when experimental designs are not available. Figure 15.1 provides an
overview of the nonexperimental designs discussed in this chapter. Let’s explore some of these options.

Figure 15.1 Overview of nonexperimental research methods

Correlational Design
As noted, there are ethical issues involved in an experimental study to assess the long-term effects of
TV violence on aggressive behavior in children. However, we suspect that you can imagine a
nonexperimental study that could assess the relationship between these two variables over the timespan of
several years. Using either a retrospective technique (examine data that already exist) or a prospective
technique (collect data across several years), you could record the degree of exposure to TV violence and
the number of aggressive incidents. Correlational research involves collecting data or searching out
records of a specified population and ascertaining the relationships among the variables of interest. Such
research involves neither random assignment nor manipulation of an experimental variable.
The two research procedures encountered most frequently, and also most sharply contrasted with each
other, are the experimental and correlational ones. Again, we repeat the important differences between
them. The experimental approach studies the causal relationship between manipulated variables and uses
random assignment (or repeated measures), whereas the correlational approach studies the relationship
between unmanipulated variables and does not use random assignment. Other examples of this approach
are: smoking history and health problems; alcohol use and GPA; education attained and salary levels, etc.
We view these research methods as complementary techniques rather than competing ones. As you shall
see, they often serve different purposes and provide answers to different questions.
 15 - 4

Random assignment of participants and the manipulation of variables are absent in correlational
research because the events of interest have already occurred or are naturally occurring. The interest is in
determining how measures on one variable are related to measures on another variable. Often, in
psychology, the two measures are behavioral measures.
The correlational approach is sometimes referred to as the study of individual differences because
emphasis is placed on differences among individuals. For example, assume that we have a distribution of
individual scores on one measure (Intelligence Test Scores—Test 1) and a distribution of individual scores
on another measure (Final Exam Scores—Test 2). The question asked of these data by a correlational
approach is whether differences among individual scores on one variable (Test 1) are related to differences
among individual scores on the other variable (Test 2).
A statistical procedure called correlational analysis is used to ascertain the extent of the relationship
among individual scores on the two variables (tests). This emphasis on individual differences contrasts with
an experimental approach where interest is in comparing the average performance of a group in one
condition with the average performance of a group in another condition (single-subject designs are an
exception).
As you may recall from your introductory statistics course, calculating a correlation between two
distributions of scores (scores on Test 1 and scores on Test 2) results in a number called a correlation
coefficient. The strength of the relationship is indicated by the numerical value of the coefficient and its
direction is indicated by a + or - sign. If the individual scores are unrelated (no relationship), the numerical
value of the coefficient is 0; if the scores are perfectly related on the two distributions, the numerical value
is either a -1.0 or a +1.0. Thus the numerical value of the correlation coefficient may range from a -1.0 to 0
or from 0 to a + 1.0, with variations in between. A positive relationship indicates that individuals scoring
high on one distribution also tend to score high on the other distribution and that those scoring low on one
tend to score low on the other. Put more simply, as individual scores on one distribution increase, their
scores on the other increase (e.g., the more one studies, the higher one’s grade point average). If the
relationship is negative, then individuals scoring high on one distribution tend to score low on the other and
those scoring low on one tend to score high on the other. Again, put simply, as individual scores on one
distribution increase, their scores on the other decrease (e.g., the more one parties, the lower one’s grade
point average).
Correlational methods are used in virtually every scientific and professional discipline and they serve
many purposes. Correlations between variables are often used to make predictions. When measures on two
variables are unrelated, i.e., correlation coefficient = 0, knowing an individual's score on one variable is not
 15 - 5

at all helpful in predicting his or her score on the other variable. As the correlation becomes greater than
zero, the accuracy of predicting the individual's score on one variable, simply by knowing his or her score
on the other, increases. And when the correlation is perfect, i.e., +1.0 or -1.0, prediction of an individual's
score on one variable from knowing his or her score on the other can be made without error.
While working on this chapter, a number of articles appeared in newspaper accounts and in popular
magazines dealing with correlational research. Some of these accounts are found in Table 15.1. They give
an idea of the variety of problems that can be studied using correlational procedures. We do not describe
the results of these reports because we have not read the primary source from which they came, nor have
we evaluated the care with which the studies were conducted. After reading the section on Third Variable
Problems and Directionality, you may want to return to this table to evaluate the extent to which these two
problems may be present in the description found in Table 15.1. You will most likely conclude that many
alternative explanations may be offered for the observed relationships.

Although correlational research allows lawful relationships to be discovered that can lead to precise
predictions, causal statements can be made only with great risk because these methods lack random
assignment, active manipulation, and rigorous control over extraneous factors. Such variables as gender,
group membership, racial characteristics, birthplace, and age are historical events over which researchers
have little control. They are determined before the researcher arrives on the scene. The measures that are
correlated are often personality variables or variables related to the characteristics of people. These
 15 - 6

variables generally cannot be manipulated. There are exceptions, of course. Some behavioral measures may
be manipulated but only with great difficulty. As we have seen, a question of ethics often arises, e.g.,
cigarette smoking and lung cancer or cardiovascular disease, exercise and cardiovascular problems.
Importance of Correlational Research
Stating that a causal relationship cannot be established by correlational research is not intended to
devalue the great importance of this type of research. It is often extraordinarily important both in a
practical and in a theoretical way. Research with this method has had a marked influence on the lives of
many people and on policy formulations of legislatures, decisions of the judiciary, and actions of private
enterprise. A few well known examples are: correlation of smoking habits with lung cancer led to a warning
on cigarette packs; the relation between exercise and lowered rates of cardiovascular problems has
stimulated increases in jogging, swimming, tennis, bicycling; correlations between socioeconomic
conditions and educational proficiency provided a rationale for decisions involving equality in education
and school busing; and correlations between particulate matter in the air (pollution) and morbidity rates
provided the impetus for clean air legislation.
It is often the case that our interests are in prediction rather than in a cause-effect analysis. After we
know the correlation between two measures, we are able to predict one form of behavior from knowledge of
the other. For example, there is a correlation between IQ test scores and success, as measured by grades in
school. Over the years considerable data have been gathered on this relationship and it is quite well estab-
lished. With a quantitative measure of the relationship (i.e., correlation coefficient), we can predict with
some accuracy success in school simply by knowing a person's score on the IQ test.
Similar examples emphasizing prediction occur when test scores are used as screening devices for
selecting those students most likely to succeed in graduate school, medical school, law school, etc. In the
latter cases, individuals take examinations over certain specified material; their scores are obtained and
prediction formulas then applied. Thus, predictions can be made on large numbers of individuals almost
immediately after obtaining their score. As we have noted, the accuracy of the prediction depends on the
strength of the relationship between the two forms of behavior. The stronger (higher) the correlation, the
better the prediction.
Discovering the relationship between two variables can be very valuable, especially if one behavior
that can be easily and inexpensively measured permits us to predict behavior that is difficult, inaccessible,
and expensive to measure. Knowing the relationship may also be of great practical value, especially for
educational systems. For example, special programs may be instituted to prevent problems from occurring
among those in need of help and to enrich those who are in need of challenge.
 15 - 7

Another example of correlational research that has proven useful relates to diagnostic purposes. After a
disorder is observed, a search can be made for other behaviors or conditions that may vary (correlate) with
it. The latter is especially the case if the disorder is difficult to detect or to diagnose accurately. If the
search is successful and the correlation strong, then both the speed and accuracy of identifying the disorder
may be substantially increased.
The usefulness and value of a correlational approach are most apparent when studying the effects of
events that simply cannot be studied in laboratory settings. This approach may be the only available
method when ethical considerations prevent manipulating the phenomena (e.g., abortion, drug use, sexual
practices, serious illness, suicide), when the phenomena are impossible to manipulate (e.g., male/female,
black/white, temperament), or when studying the effects of natural disasters such as earthquakes, fires,
violent storms, etc.
Direction of Control and Third Variable Problems
Discussion of direction of control and third variable problems will illustrate the difficulties of inferring
cause-effect when interpreting correlational data. With correlational research, we usually refer to predictor
and criterion variables rather than independent and dependent variables. The measure (or behavior) being
predicted is the criterion variable, and the measure (or behavior) from which the prediction is made is the
predictor variable. The use of this terminology emphasizes prediction rather than suggesting a cause-effect
relationship. However, there are occasions when individuals, be they scientists, writers, or laypeople, come
to cause-effect conclusions based on correlational data. They face a risk of drawing false conclusions when
doing so. In effect, they must deal with two different problems: the direction of control problem and the
third variable problem. We will give examples of both.
To infer a cause-effect relationship requires that we specify the direction of control. Assume that
variable X and variable Y are highly correlated such that increases in one are associated with increases in
the other. Does variable X cause variable Y to vary, or does variable Y cause variable X to vary? With
some relationships the answer concerning the direction of control seems obvious but in other instances it
can be difficult to specify. Let’s assume that our correlational research on TV violence and aggression
showed a positive relationship between these two variables. It would seem to be a natural inference to
conclude that exposure to TV violence leads to (causes) more aggressive behavior in children. However, is
this necessarily the case? Are there alternative explanations for the observed relationship? Is it not possible
that children who are more aggressive, for whatever reason, tend to choose TV shows with more violence?
We will give another example of the direction of control problem. Let us say that a high, positive
correlation exists between frequency of drug use (variable X) and difficulties in school (variable Y). We
 15 - 8

could say that the use of drugs was the cause of experiencing difficulties in school. On the other hand, an
equally plausible conclusion is that having difficulties in school caused the individual to use drugs.
The fact that in both of these examples we have a high correlation between variable X and variable Y
does not help at all in our determining the direction of control. Our risk of coming to a wrong causal
conclusion is not reduced. The only way to reduce the risk is to bring additional information to bear on the
issue or, when permitted, to attempt an experimental approach and manipulate the important variables.
Our problem is intensified in that the risk of error is even greater when we consider possible third
variable problems that may be present. For our example with the TV violence study, it is possible that
neither of the two variables causes the other. Rather, some other (third) variable may actually cause
changes in the two variables that were measured. After giving this some thought, we are sure that you can
come up with a potential third variable. In our example that involved drug use, instead of drug use causing
school problems, or school problems causing drug use, a third factor (fourth, fifth, etc.) could have caused
increases in both drug use and school problems. For example, anxiety, depression, low self-esteem, or
conflict within the home could give rise to both increases in drug use and increases in school problems. In
this case, then, variables X and Y are related only through some third variable. Figure 15.2 illustrates both
the direction of control and third variable issues.
 15 - 9

Figure 15.2 Direction of control and third variable issues in correlational research

It is sometimes difficult to avoid a causal conclusion and to maintain the proper perspective when
viewing correlational data. The consistent, highly publicized, positive correlation between cigarette
smoking and cancer seems causal. The more cigarettes smoked, the higher the incidence of cancer. This
relationship seems to suggest that the smoking of cigarettes controls whether or not cancer will occur. Yet,
strictly speaking, we cannot say this based on human research. A third factor may be involved. It is
possible that only certain people with this factor smoke and that these people would contract the disease
whether or not they did so. The more of this factor present the greater would be smoking and the possibility
of contracting the disease. If this were so, then increases in both smoking and cancer result from this un-
specified third factor. The latter is very unlikely and, in fact, the few recent experimental studies that have
been done suggest that it is certain ingredients in tobacco that may cause cancer. Third variable problems
are sometimes very difficult to detect and to deal with. Further, you can never be sure that they are absent
when using correlational designs. However, some techniques are available for addressing the problem.
Addressing Directionality and Third Variable Problems
There are several methods available that attempt to deal with the problems of directionality and third
variables. Although these methods are sometimes useful they do not solve the preceding two problems.
Each has its problems. A full description of these methods is beyond the scope of this text. We mention
them here for those of you who would like to pursue the problems further. A procedure that is sometimes
useful for addressing the directionality problem is a time-lagged correlational design. The essential aspect
of this procedure is to determine a correlation between two variables at Time 1 and again later at Time 2.
This technique is also referred to as cross-lagged panel correlation. The third variable problem has been
addressed by some using a correlational procedure referred to as partial correlation. In this case, if a third
variable is thought to be a factor affecting the correlation between variables X and Y, the partial correlation
technique can be used to eliminate its effects. The result of this procedure is a correlation between X and Y
uninfluenced by the third variable. Information regarding these procedures can be found in more advanced
research design/statistics textbooks. Another procedure that is sometimes used to eliminate the third
variable problem is matching. In this case only data from participants matched (who are the same) on the
third variable are analyzed.
For example, if we suspect that the parenting style is a third variable that affects both the amount of
TV violence watched by the children and the aggressiveness of the children, we may want to include only
those children in the study who have parents with similar parenting styles. At first glance, matching as a
 15 - 10

technique for ruling out third variable interpretations seems like an attractive solution. That is, if only
individuals who were the same or highly similar on these third variables were assessed, and if the
relationship between the two variables of interest still existed, then the third variable considerations could
be dispensed with. The thought is a good one, but in practice problems exist; also, matching can be difficult
to achieve. One problem is that researchers cannot be sure they have considered the relevant third variables.
Some may exist that have not been considered. Another problem is that matching on one variable can
sometimes unmatch individuals on other variables. Researchers face a more serious problem when they
attempt to match on several variables (e.g., intelligence, education, motivation, class). The problem is
simply in getting a sufficiently large sample that has the necessary characteristics.
Correlational Ruling Out Factors
The correlation between cigarette smoking and cancer (and heart disease) can be used to illustrate an
important aspect of correlational research. Had no relationship been found, we could have ruled out causal
factors and there would be little interest in pursuing the problem further. Therefore, an important con-
tribution of correlational methods may be in a negative sense: In the absence of a relationship, there is no
need to devote time to an experimental analysis of the problem to identify the controlling or causal factor.
Perhaps an example contrasting the correlational method with the experimental method would be
helpful in illustrating why a causal relationship can be made with an experimental approach and not a
correlational one. As noted, in a correlational study involving cigarette consumption and incidence of
cancer, a positive relationship was found. While most people of sound mind would be alerted to a possible
causal link between the two, a strong relationship regarding cause cannot be made. We indicated that it is
possible that a third factor may be the cause, e.g., people susceptible to cancer also smoke, but the disease
would occur whether or not they smoked. Or we could argue that individuals who smoke also engage in
other activities that may be related to cancer and that smoking is not the problem. Or, we could say that
individuals with certain dietary habits are susceptible to both smoking and cancer, etc.
The tobacco industry today argues that a causal link has not been clearly established in humans. To
decide the question, an experiment is necessary. It would not be difficult to design a research project to
answer the question of smoking as a cause for illness. However, serious problems of ethics and practicality
prevent its implementation. We would randomly select a large number of twelve- to fourteen-year-old male
and female students from different geographic areas. We could then randomly assign them to conditions A,
B, C, D, and E—corresponding to levels 0, 1/2, 1, 2, and 3 packs of cigarettes a day. We would then
follow them up over the years with annual physical examinations and also record specific measures known
to be related to tissue problems. We could then determine whether there was a systematic relationship
between our independent variable (smoking) and our dependent variable (measures of illness, etc.).
 15 - 11

Obviously, the study cannot and should not be done. However, experimental studies have been conducted
with nonhuman species.
Mice, rats, and dogs have been used to study the relationship between the exposure to tobacco
ingredients and cancer. A variety of procedures have been used, such as placing the substances in tobacco
on the skin, confining the animals to enclosed rooms where controlled amount of cigarette smoke may be
dispersed, and teaching animals to smoke. These studies have established that tobacco is hazardous to a
laboratory animal's health. We have stated several times that it is improper to draw cause-effect
conclusions from correlational data. To say this is not to say that such a relationship does not exist. It may.
To determine its existence requires other research strategies.
There are many examples of correlational studies reported in the media that falsely suggest a cause-
effect relationship. You are probably familiar with reports of a link between the phase of the moon and
“strange” behavior. Box 15.1 illustrates one such report.
 15 - 12

Box 15.1 Thinking Critically About Everyday Information – Does the full moon make you crazy?
The following report was obtained from the online edition of Weekly World News. The title of the report is
“Research Proves Full Moon Really Does Make You Crazy!” The report reads: “Amazing research proves
that the full moon really does drive people loony. And it can even make them kill! Scientific tests confirm
that murder rates triple around the time the moon is full. And suicide rates also peak during the full moon
of spring - and the new moon in autumn and winter. ‘Human biological tides respond to the moon's pull
much as the sea does and for a very similar reason,’ Dr. Arnold L. Lieber, author of The Lunar Effect
(Anchor Press) told a reporter. I found that in my studies of violent behavior in U.S. cities, instances of
murder, aggravated assault, suicide, psychiatric emergencies and fatal auto accidents all went up during the
full moon.’ Dr. Lieber first noticed the curious lunar effect when he was a medical student working in the
Jackson Memorial Hospital in Miami and realized that mental patients became highly disturbed during the
first few days of each month. And when he approached senior doctors about the phenomena, they told him
that patients suffered more epileptic seizures and bleeding ulcers during the full moon. Then police
confirmed that criminal behavior was also affected - producing graphs showing that violence and arson
increased during the full moon. But Florida police aren't the only ones who have noticed the strange effect
of the lunar cycle on criminal behavior. Officers with just about every police department around the globe
can tell stories about "mooners," the weirdos who misbehave when the moon is full. And occultists refer to
the period of waning lunar phases as a "sacrificial moon," researchers point out. The first victims of serial
killer Ted Bundy are known to have been murdered in Washington state during the waning phases of the
moon - always under Taurus, Scorpio or Pisces. DeWitt Clinton Cook became known as "The Moon
Maniac" after attacking numerous women and committing 300 burglaries - all while there was a full moon.
Cook was executed at California's San Quentin Prison in 1941 for the murder of a college coed. And
Robert Louis Stevenson's classic study, Dr. Jekyll and Mr. Hyde, was patterned after the true story of a
lunatic who committed horrible crimes during the full moon.

Consider the following questions:

1. Although some of the evidence cited in the article is anecdotal, some is based on actual crime statistics.
Do these latter analyses represent experimental or nonexperimental designs?

2. The article suggests that the moon phases cause a change in our biology. For the relationship between
moon phase and criminal behavior, could there be a direction of control problem?

3. For the relationship between moon phase and criminal behavior, could there be a third variable
problem? If so, what is a possible third variable that would explain the relationship?

4. The title of the article states that “Research proves…” What is wrong with this?

5. The article clearly states that there is a causal link between moon phase and “crazy” behavior. How
would you respond?

Retrieved June 10, 2003 online at:

http://www.weeklyworldnews.com/bizarre/bizarre.cfm?instanceid=13224

Interpretation of Correlational Data

Now that we understand the nature and value of correlational research, let’s explore a specific
example and discuss some of the issues involved in the interpretation of correlational data. We’ll return to a
research idea mentioned earlier in this chapter, namely the long-term effect of TV violence on aggressive
 15 - 13

behavior in children. As we noted, random assignment of children to groups that view specific levels of TV
violence for several years is not ethical. However, we could conduct a correlational study in which children
(with assistance from parents) maintain a weekly log of TV shows watched from the time that they are 10
years old until they are 15 years old. Based on these logs, we calculate the mean number of hours of TV
violence viewed per week. For the variable that measures aggressive behavior, we obtain the number of
disciplinary incidents recorded at their school for the same five-year period. Hypothetical data from 20
students are shown in Table 15.2.
 15 - 14

Glancing through the pairs of scores in Table 15.2, whether there is any systematic pattern in the data
is difficult to detect, although the one participant with 25 hours/week and 30 incidents does stand out. With
correlational data, it is not only helpful, but critical, that the data be graphed in a scatterplot. Figure 15.3 is
a scatterplot of the data such that a point on the graph represents each pair of scores. Because the values on
the x-axis increase from left to right and the values on the y-axis increase from bottom to top, it is now
easy to detect a general pattern in the data such that as the mean hours of TV violence per week increase,
the number of disciplinary incidents at school also increases. The presence of the one outlier is also clear on
the graph. Because our data are measured on a ratio scale and because our graph suggests a linear
relationship, we can calculate a Pearson r correlation coefficient to quantify the relationship (see Figure
10.10 to review the decision tree for measures of relationship). The calculation results in r = .88, a
significant positive correlation.

Figure 15.3 Scatterplot that depicts the hypothetical data in Table 15.2

It is important to be aware in correlational research that outliers in the data can have dramatic effects
on the value of the correlation coefficient and, therefore, the conclusions drawn. For example, if we remove
the one outlier point from our hypothetical data, the r value shifts from .88 to .62. Thus, that one
participant dramatically increased the strength of the relationship between the two variables. In other cases,
the outlier can be such that it dramatically decreases the strength of the relationship. In either case,
researchers should always carefully inspect the data that produced outliers to be sure that no errors were
 15 - 15

made in observations or calculations. Even without errors, in some cases there is something unique about
that participant that may warrant their removal from the data set. If data is removed from the data set, the
rationale must be clear and must be reported. A researcher should never remove data simply because it is
different from other data or because it works against the hypothesis for the study.
A few other issues that deserve mention are restriction of range, curvilinear relationships, and sample
size. Restriction of range refers to the fact that when the range of scores on one or both of your variables is
small or nonexistent, you have not adequately addressed the research question and you will almost certainly
obtain a nonsignificant value for the correlation coefficient. For example, if all 20 participants in our
hypothetical study had watched a mean of 2 hours of TV violence per week, the correlation coefficient
would be 0.00 (regardless of the values for the number of disciplinary incidents). We would not have
addressed the research question. Therefore, when you select variables for correlational research, it is
imperative that the measurement of those variables result in a relatively wide range of values from the
participants.
It is also important to examine the nature of the relationship between the two variables to determine
the most appropriate correlation coefficient. In our analysis of the hypothetical data in Table 15.2, the
scatterplot suggested a linear relationship. That is, the form of a best-fitting line through the points on the
scatterplot was a straight line. In some situations, the relationship between two variables may be monotonic
but curvilinear. That is, the form of a best-fitting line through the points on the scatterplot is a curved line.
Consider the scatterplot shown in Figure 15.4. You would probably agree that there is a relationship
between mean hours of TV violence per week and number of disciplinary incidents. However, it is a
curvilinear relationship rather than a linear relationship. Thus, the Spearman correlation coefficient is the
proper statistic rather than the Pearson correlation coefficient (again, refer to the decision tree in Figure
10.10).
 15 - 16

Figure 15.4 Scatterplot that illustrates a curvilinear relationship

Finally, sample size is an issue to consider in correlational research. The power of a correlational
analysis is increased by increasing the sample size. With extremely large samples, weak relationships with
correlations of .1 or .2 may be statistically significant but not meaningful practically. On the other hand
with a small sample size a meaningful relationship might go undetected. The concept of power is an
important one (see Chapter 10). Although correlational research represents a powerful nonexperimental
design, it is time to turn to several other types of nonexperimental designs.

Ex Post Facto Design

At times, caution must be exercised in deciding whether a study is nonexperimental or experimental. If
the determination is not made, an erroneous conclusion may result. A type of study that can masquerade as
a genuine experiment is the ex post facto design. Although it appears to be a true experiment because of
the way groups are separated and the way the analysis is performed, it is still nonexperimental research and
subject to the same limitations discussed earlier with correlational research. It derives its name from the
fact that the assignment of participants to levels of the independent variable is based on events that
occurred in the past (i.e., after the fact). It mimics an experiment in that comparisons are made between
two or more groups of individuals with similar backgrounds who were exposed to different conditions as a
result of their natural histories. We then measure the participants on a dependent variable of interest to
 15 - 17

determine whether or not statistically meaningful differences exist between the experimental groups. If
reliable differences are found, should we conclude that they were due to the historical differences we found
in the past records? The answer is no.
Note that the ex post facto design uses neither random assignment nor active manipulation of the
independent variable. However, the intent of this type of research is precisely that of a true experiment but
the problems encountered in drawing conclusions are very different. A few examples will illustrate our
point.
In Chapter 11, we considered an experimental design in which we, as the researchers, determined which
levels of TV violence would be studied and randomly assigned research participants to these levels. We
then measured aggressive behavior. A nonexperimental alternative would be to place participants into
groups based on how much TV violence they just watched (of their own choosing) and to then measure
aggressive behavior. Notice that these two research designs are similar in that the aggressive behavior of
groups of participants are being compared, in which the groups differed in terms of exposure to TV
violence. We should also mention that the statistical analysis would be similar. However, notice the
differences. The latter design is an ex post facto design because participants were not randomly assigned to
the groups, i.e., the independent variable was not manipulated by the researcher. Because we lack the
control of an experimental design, we must be very cautious with the nature of the conclusion that we draw
from the ex post facto design. As we noted with correlational research, direction of control and third
variable issues need to be considered. Let’s examine another example.
An instructor in a college math course believes that a relationship exists between performance in
college math courses and whether students had the "old math" or "new math" techniques in grade school.
She decides to do a study to determine whether her observations are indeed accurate ones. She looks up the
grade school records of a large number of college students taking her college math course. On the basis of
these records she selects fifty students who received "new math" and another fifty students who received
"old math" techniques. She then gathers the two groups of students together and gives them a college math
proficiency test. Her experimental design would look no different from that of a true experiment.
We have what appears to be a "treatment" condition and a comparison or control group (old math)—in
this case both of which have already occurred. The “treatment versus comparison group” appears to be the
same comparison characterized by a true experiment. But any conclusions drawn from our example must
be guarded and weak. Why? Obviously, our participants cannot be assigned randomly, (nor can we
manipulate the conditions because we have selected students already). Their assignment to the levels of the
independent variable was based on past records.
 15 - 18

The groups may be biased in different and unknown ways. It is quite possible that those who received
new math in grade school are very different from those who received the old math. They may have been
chosen because of their skills (or lack of skills) in mathematics. It is also possible that new math programs
were offered only by wealthy and progressive school districts. Thus, it is quite possible differences existed
in the student populations from which the treatment groups were drawn. Perhaps only urban-industrialized
communities had new math programs but rural-suburban communities did not. The point is that this is all
idle speculation. We simply do not know the many reasons that some students received new math and some
old math. Because we could not randomly assign our students to the two groups, systematic participant
bias is a distinct possibility. Our two student groups could have differed in their average math proficiency
score for any number of reasons other than "new" or "old" math. (A true experiment would have required
that we previously randomly assign a number of students from the same school district to new math and
approximately an equal number to old math classes, holding other factors constant. Then, years later, we
would administer a math proficiency test in college.)
Two questions, if asked, could quickly and accurately allow you to identify whether the experiment is
an ex post facto one: (1) The key consideration is did the researcher have control over assigning
participants randomly? That is, did the researcher assign participants randomly to groups? (2) Second, did
the researcher actively manipulate the independent variable or had it already occurred? If random
assignment was not used then it was not a true experiment. Now the choice is deciding between a
quasi-experiment or an ex post facto one. If a variable was not actively manipulated at the time of the
research, then the experiment is an ex post facto one and the conclusion must be much weaker. Keep in
mind that in ex post facto research those that make up the groups in itself constitute the "treatments." There
is no treatment other than membership in a group, e.g., male versus female, heavy versus thin, tall versus
short, joggers versus nonjoggers, dogmatic versus nondogmatic, smokers vs. nonsmokers, etc. A quasi-
experiment, lacks random assignment but it does have control over levels of the IV.
Another example of an ex post facto experiment will help you to discriminate between it and
experiments in which the independent variable is, in fact, manipulated. In this example our interests are in
the reactions of anxious and nonanxious participants to a loud noise. Do not confuse the loud noise with the
"treatment" condition. It is not a treatment or independent variable, since both groups receive it. It is simply
a task to which all participants respond. The reaction to the noise is our dependent variable measure, i.e.,
the task that all participants perform. The treatment consists of membership in a group based on whatever
characteristics the researcher may be interested in. In the present example these characteristics are anxious
and nonanxious participants. Let's get on with the example.
 15 - 19

We could give a personality test (e.g., the Taylor Manifest Anxiety Scale) to a large population of
students and then select participants on the basis of their scores. We might select ten students who score
very high and ten students who score very low on the anxiety scale. Since high anxious and low anxious
participants are our interest and, as it were, our treatment, they cannot be randomly assigned to different
groups. Therefore, our independent variable is high versus low anxiety. Our dependent variable is the
magnitude of the startle reaction to loud noise. Assume that we find our high anxious participants respond
to noise more strongly than do our low anxious participants. Can we conclude that they did so because of
anxiety? No, we cannot. We are again plagued by the third variable problem. Because we could not employ
random assignment, i.e., our groups were pre-existing groups, it is possible that our participant groups
differ in many ways other than anxiety. Their responses could be related to how well they sleep. It may be
that high anxious participants sleep less well than others and that any participant, regardless of anxiety,
who did not get adequate sleep would perform like high anxious participants. Perhaps it is the case that
high anxious participants drink much more coffee than do low anxious participants. If we assume that
coffee affects the startle response, then any participants who drink much coffee, anxious or nonanxious,
would perform in similar ways. There are any number of other alternative interpretations allowed by this ex
post facto design that would not be possible if the research was either a quasi- or a true experiment.

Naturalistic Observation
Naturalistic observations take place under natural conditions or under real-life conditions without
any intervention on the part of the researcher. Such observations contrast with those in experimental set-
tings where considerable prior control is possible, where events can be manipulated, and the observations
may be repeated. When using naturalistic observation, we observe nature without imposing change.
However, the observations are carefully planned and systematized. Consequently, the data can be organized
in a meaningful way to permit analysis and interpretation. However, the interpretation issues already
discussed in this chapter must be considered.
Naturalistic observation is the oldest method for the study of behavior or other phenomenon. One of
the most accurate sciences (in terms of prediction) is astronomy and it is restricted largely to observation.
Ethologists are people who study animal behavior under natural conditions and use natural observation
almost exclusively. Jane Goodall's work with primates in Africa has resulted in some fascinating
observations regarding their social interactions. Development and personality psychologists have published
extremely informative observational studies of children interacting under natural conditions. Jean Piaget's
theory of cognitive development in children was based on naturalistic observation. We have been able to
 15 - 20

identify migration patterns in fish, fowl, and mammals by tagging studies. Naturalistic observation of
primitive cultures has given us insight into the range of variation in human institutions. It must be
emphasized that natural observation is not anecdotal and casual but, instead, systematic and carefully
planned. The observer must be sufficiently skilled to distinguish between an observation and an
interpretation. Many of the principles of good observation were discussed in Chapter 6.

Qualitative Research
All of the research designs discussed thus far have involved the measurement of variables with a
subsequent statistical analysis of the values recorded. The application of mathematics to the process of
making research conclusions lends a degree of objectivity to the decision-making process. Although there
can be differences of opinion regarding the most appropriate statistical analysis, for the most part, a given
set of data would lead different researchers to the same conclusion. There is value to this but there is also
value to qualitative research, in which the researcher gains insights from more informal and non-numerical
observations. In fact, conclusions drawn from such informal observations are often a first step that leads to
more rigorous experimental studies to clarify relationships among variables. A good researcher understands
that the various types of experimental and nonexperimental designs complement one another when a
particular area of behavioral research is studied.
Case Study Design
With the case study design, one or more individuals are carefully examined over time. Biographical
data, interviews, or psychological tests may be components of the case histories. Some examples of case
histories are prepared by reconstructing the biography of the individual from memory and records
(retrospective). On occasion, a case history may be prospective. It does not rely on memory or records
because measurement is taken at periodically planned intervals.
There are two important considerations in a case study approach. One is to search for some regularity
or patterning to behavior that might suggest some principle around which it is organized. The second
involves additional case studies to confirm the previously observed regularity or pattern. Such information
may permit generalization to other situations or persons.
Case studies often arise when it is impossible or unethical to conduct an experiment. At other times,
they are closely related to naturalistic observations. Freud's insights into behavior problems were largely
based on case studies of his patients. Piaget's theory of intellectual development stems from his intensive
observations of his own three children. Case studies of individuals suffering brain damage have been
undertaken to assess the extent to which functions are regained.
 15 - 21

Phenomenology
How did the surviving New York City firemen process the events that happened at the World Trade
Center on September 11, 2001? Understanding their experiences may provide insight into how people deal
with traumatic events. An appropriate research design would involve a phenomenological approach. With
phenomenology, the researcher seeks an in-depth interview with an individual or a group of individuals to
understand their experience. The experience may relate to some traumatic event (September 11, death,
natural disaster), a joyous event (wedding, birth, winning a sports championship), or any other
distinguishing event. The research participants are asked to introspect – to describe their feelings and how
they explain the event and their actions. Based on the interview(s), the researcher attempts to discern some
underlying principle, conclusion, or generality. There is, of course, a degree of subjectivity and
interpretation on the part of both the participants and the researcher. However, it would be rather difficult
to study this aspect of the human experience with other research designs.
Ethnography
What could motivate the Al Qaeda terrorists to commit such a violent act as the September 11 attack?
To answer this question, one needs to fully understand the way of thinking and the way of life of a group of
people. Ethnography is an approach designed to understand a group of people that form a culture. The
culture to be studied may be quite large (e.g., Asians, Black-Americans) or quite small (e.g., African tribe,
inner-city gang, inmates in a prison).
Information for an ethnographic study can be gathered in several ways. Similar to the
phenomenological approach, members of the culture can be interviewed. In addition, artifacts produced by
the culture (books, art, music, photographs) can be analyzed. Finally, the researcher may simply observe
the culture. As discussed in Chapter 6, such observation can involve being a participant observer (observe
from the inside) or a nonparticipant observer (observe from the outside) and the observations may or may
not be known to the members of the culture. As with all qualitative research, a certain degree of subjectivity
and interpretation is inherent to the ethnographic approach.

Case Analysis
In 1998, the Psi Chi Journal of Undergraduate Research published a study entitled “The Effects of
Extracurricular Activities on Self-esteem, Academic Achievement, and Aggression in College Students.”
The researchers (Bleeker, Evans, Fisher, & Miller) administered a set of questionnaires to 113
undergraduate students and concluded “the scores on these inventories were related to the students’
 15 - 22

extracurricular involvement and their cumulative grade point averages (GPAs). Women involved in social
clubs had significantly higher self-esteem scores than women involved in athletics and women not involved
in activities. Men participating in social clubs had significantly higher hostile aggression scores than did
women involved in social clubs. Thus, for women, self-esteem is positively related to extracurricular
activities, whereas for men, the relationship is less clear.” Let’s assume that the Director of Student
Services on your campus uses these findings to suggest that women should get involved in social clubs to
increase their self-esteem.
Critical Thinking Questions
1. What type of nonexperimental research design was used in this study?
2. What is wrong with the suggestion by the Director of Student Services? Why is it wrong?
3. Is “direction of control” a potential issue in this study? Explain.
4. Is there a potential “third variable” issue in this study? If so, provide an example.
5. Look again at the title of the article. There is something wrong with it. Re-write the title so that it better
conforms to the type of research design that was used.

General Summary
In addition to experimental and quasi-experimental research designs, nonexperimental designs offer
valuable techniques for better understanding human behavior. As a category, nonexperimental designs are
characterized by a low level of experimental control, lack of cause-effect conclusions, and an increased
likelihood that the data were obtained from “real-world” settings. Correlational research assesses the nature
of the relationship between predictor and criterion variables, while ex post facto research assesses the
differences between pre-existing groups on some dependent variable. Conclusions should only be drawn
after careful consideration of direction of control and third variable issues. Naturalistic observation is a
method in which observations occur in a natural setting without any interference by the researcher. In
addition to research designs that are based on quantitative measurements, qualitative techniques can
provide important information. Case studies involve a careful analysis of an individual or small group of
individuals over time, phenomenological studies rely on in-depth interviews to better understand the
subjective experience of individuals, and ethnographic studies seek to better understand the behavior of
individuals who are part of a particular culture. Now that we have explored a variety of experimental and
nonexperimental designs, we turn in the next chapter to methods by which we share our research with the
public.
 15 - 23

Detailed Summary
1. Nonexperimental designs are characterized by the researcher’s lack of control over the independent
variable.
2. Although experimental designs are quite powerful, there are several good reasons to use
nonexperimental designs. There are many interesting research questions for which the researcher does
not manipulate or control the independent variable due to practical, ethical, or financial reasons.
3. Correlational research involves collecting data or searching out records of a specified population and
ascertaining the relationships among the variables of interest. Such research involves neither random
assignment nor manipulation of an experimental variable.
4. Experimental designs study the causal relationship between manipulated variables and use random
assignment (or repeated measures), while correlational designs study the relationship between
unmanipulated variables and do not use random assignment.
5. The degree of relationship between variables in a correlational design is measured by calculating a
correlation coefficient.
6. Correlation coefficients have values that range from –1.0 to +1.0. Positive correlations indicate that as
values for one variable increase, values for the other variable also increase. Negative correlations
indicate that as values for one variable increase, values for the other variable decrease. Zero
correlations indicate no relationship between the variables.
7. Correlations are used to make predictions regarding behavior. Such predictions can be used to make
decisions about who gets selected for a job, a special program, admission to college, etc. Correlations
are also used to assess the reliability of testing instruments and observers. Other uses include
assessment of test validity, development of diagnostic criteria, and theory evaluation.
8. The issues of direction of control and third variables are two reasons why cause-effect conclusions are
unwarranted in correlational research.
9. Even if two variables are correlated, the correlational analysis does not reveal which variable is causing
changes in the other variable. This is the direction of control problem.
10. When two variables are correlated it is possible that some third, unmeasured variable is responsible for
the relationship because that third variable is simultaneously causing changes in the original two
variables. This is the third variable problem.
 15 - 24

11. A procedure that is sometimes useful for addressing the directionality problem is a time-lagged
correlational design. The essential aspect of this procedure is to determine a correlation between two
variables at Time 1 and again later at Time 2.
12. The third variable problem has been addressed by some using a correlational procedure referred to as
partial correlation. In this case, if a third variable is thought to be a factor affecting the correlation
between variables X and Y, the partial correlation technique can be used to eliminate its effects.
Another procedure that is sometimes used to eliminate the third variable problem is matching. In this
case only data from participants matched on the third variable are analyzed.
13. Although correlation does not imply causation, the absence of a correlation does imply the absence of a
causal relationship.
14. The interpretation of data from correlational research requires a scatterplot to detect the form of the
relationship (linear vs. curvilinear), the presence of outliers, and to evaluate variables for restriction of
range. The form of the relationship determines the appropriate correlation coefficient and either of the
other two occurrences can have a dramatic effect on the value of the correlation coefficient and the
interpretation of the relationship.
15. The ex post facto research design mimics an experiment in that comparisons are made between two or
more groups of individuals who were exposed to different conditions as a result of their natural
histories. We then measure the participants on a dependent variable of interest to determine whether or
not statistically meaningful differences exist between the experimental groups.
16. Because the ex post facto design uses neither random assignment nor active manipulation of the
independent variable, causal conclusions are not warranted.
17. When observations are made in a real-world setting, we refer to these studies as naturalistic
observations.
18. Qualitative research occurs when the researcher gains insights from more informal and non-numerical
observations.
19. A case study involves the systematic collection of information about one or a few individuals either as
historical data (retrospective) or as data collected over time (prospective).
20. With phenomenology, the researcher seeks an in-depth interview with an individual or a group of
individuals to understand their experience.
21. Ethnography is an approach designed to understand a group of people that form a culture by
interviews, artifacts, and direct observation.
 15 - 25

Key Terms
Case study
Correlation coefficient
Correlational research
Criterion variable
Direction of control
Ethnography
Ex post facto design
Matching
Naturalistic observation
Phenomenology
Predictor variable
Qualitative research
Third variable

Review Questions/Exercises
1. In this chapter, we suggested several research questions that would be appropriate for each of the
nonexperimental designs. Develop your own research question for a correlational design, an ex post
facto design, a naturalistic observation, a case study, a phenomenological study, and an ethnographic
study. Briefly describe the methodology for each.
2. Summarize the distinctions between experimental and nonexperimental research and also between
quantitative research and qualitative research.
3. There has been considerable concern about whether the scholastic performance of public school
children has declined. Changes in the schools have occurred and there is some evidence that structure,
discipline, and supervision have decreased in the public school system. Some educators have argued
that the latter are the reasons for the decline in academic performance. To support their view they
undertook a study comparing the scholastic performance of students in parochial schools (with
documented high levels of supervision, discipline, structure) with those in public schools. Their beliefs
were confirmed. The data revealed that students from parochial schools significantly out-performed
students from public schools on each measure of performance. Based on these data they made a
concerted effort to increase discipline, structure, and supervision in the public school system. What
type of research design was used? Why would you be critical of their response to the study?
 15 - 26

4. Alcohol is the drug of choice on most college campuses. Studies show that students who drink more
alcohol per week have lower grade point averages. Many educators and student personnel
administrators have argued that since alcohol consumption causes lower GPAs, anti-drinking programs
should be immediately instituted to enhance education. What is wrong with their reasoning (include a
discussion of third variables in your answer)?