Method

Download as pdf or txt
Download as pdf or txt
You are on page 1of 19

Copyright American Psychological Association

CHAPTER 1

METHODOLOGY: WHAT
IT IS AND WHY IT IS SO
IMPORTANT
Alan E. Kazdin

Scientific knowledge is very special. This knowledge components to convey the scope of the topics and to
is based on the accumulation of empirical evidence organize the subject matter.
and obtained through systematic and careful obser-
1. Research design: This component refers to the
vation of the phenomenon of interest. At a very
experimental arrangement or plan used to examine
general level, the ways in which the observations
the question or hypotheses of interest. It includes
are obtained constitute the methods of science. Yet,
fundamental issues related to who the participants
these methods can be considered at multiple levels,
will be, how they will be assigned (e.g., randomly),
including the principles and tenets they are designed
and the comparisons (various groups) included
to reflect, a way of thinking and problem solving,
and concrete practices that scientists use when actu- in the study. Many different arrangements exist,
ally conducting an investigation. This book draws including those in which some experimental
on each of these levels because they work together manipulation is made (true experiments) or
and make for good science and scientific research. groups are formed (observational study), by which
The purpose of this introductory chapter is to to evaluate differences in characteristics of interest.
convey what methodology is, why it is needed, and 2. Assessment: This component pertains to the
the key tenets that guide what we do as scientists. measurement strategies (e.g., self-report, neuro-
These foci may seem obvious—after all, everyone imaging) and the measures that will be used to
knows what methodology is and why it is needed. provide the data. There are many different types
Perhaps so, but the answers are not all so obvious. of measures and multiple measures within each
It is useful to give the rationale for what we do and type. Key issues related to assessment, such as
why because it provides the common base we as reliability and validity of the measures, are
psychologists and social scientists share with all pivotal to research.
of the sciences. Also, that base underpins all of the 3. Data evaluation and interpretation: This compo-
chapters that follow. Let us begin. nent encompasses all of the methods that will be
used to handle the data—to characterize the
sample, to describe performance on the
SCIENTIFIC METHODOLOGY
measures, and to draw inferences related to the
AND ITS COMPONENTS
hypotheses. Statistical significance testing is
Methodology in science refers to the diverse prin- dominant and the most familiar method used to
ciples, procedures, and practices that govern empiri- develop and evaluate data but, as later chapters
cal research. It is useful to distinguish five major show, other methods are also used.

http://dx.doi.org/10.1037/14805-001
Methodological Issues and Strategies in Clinical Research, Fourth Edition, A. E. Kazdin (Editor)
Copyright © 2016 by the American Psychological Association. All rights reserved.

3
Alan E. Kazdin Copyright American Psychological Association

4. Ethical issues and scientific integrity: This multi- cal models and analyses have their own courses and
faceted component includes a variety of respon- journals. This book covers all five components and
sibilities that the investigator has in the conduct does so in a way that underscores their integration
of the study and can encompass all of the other and interrelation. There are always more topics and
components (e.g., design, data analyses, publica- components of methodology one could add. For
tion of findings). Ethical issues include multiple example, the historical roots of science and science
responsibilities to participants (e.g., their rights and social policy are legitimate topics that could be
and protections) and adherence to the profes- covered as well. Yet, in developing an appreciation
sional standards of one’s discipline (e.g., ethical for methodology and the skills involved in many of
codes). Scientific integrity includes responsibili- the key facets of actually conducting research, the
ties to the scientific community and the public five will suffice.
at large (e.g., transparency, accurately reporting
findings) and is also part of professional standards WHY DO WE NEED SCIENCE AND
and ethical codes. Before a study begins, propos- ITS METHODS AT ALL?
als are usually required (e.g., by universities,
agencies) that discuss not only specifics of the Rationale
project (e.g., research design, assessment) but I have already mentioned the components of
also ethical issues and assurances that participant scientific methods, but now let us step back a bit.
rights are protected (e.g., scrutiny of the proce- Why do we even need methodology in general
dures for any untoward effects, informed consent, and its components? Four reasons can make the
protection of privacy). case for why we need science and the methodology
5. Communication of research findings: of science. First, we need consistent methods for
Communication of our work is key to building acquiring knowledge. There are many sciences,
the knowledge base, stimulating responses to our and it would be valuable, if not essential, to have
work, and promoting and fostering new theory principles and practices that are consistent across
and findings as we ourselves or others follow up them all. We would not want the criteria for what
on the study we have described. Findings can be counts as knowledge to vary as a function of quite
different ways of going about obtaining that knowl-
communicated to other professionals through
edge. This consistency is more important than
many different venues (e.g., journal articles on
ever today, because much of research on a given
empirical studies, review articles, conference
topic involves the collaboration of scientists from
symposium presentations, poster sessions).
many different fields and many different countries
Communication also includes the media
to address a set of questions for a given project.
(dissemination of information to the public via
Scientists from many different areas must speak the
TV, radio, and the web). Communication of
same methodology language, share the same under-
findings has its own responsibilities and chal-
lying values about how to obtain knowledge, and
lenges, as discussed later.
agree on procedures and practices (e.g., statistical
I have divided methodology into these compo- evaluation, reporting data that do and do not sup-
nents in part to convey the breadth and depth of port a particular hypothesis). Consistency is also
the topic. There are books, courses, and journals critical within any given scientific discipline. For
devoted specifically to each of these components. a given science (e.g., psychology), we would want
As one example, psychological assessment is an consistency throughout the world in the standards
enormous topic encompassing models of scale for obtaining scientific knowledge—the accumula-
development, validation, the vast range of assess- tion of knowledge from all individuals in a given
ment modalities, and sources of artifact and bias field requires this level of consistency. Science says,
that can greatly affect data obtained from a measure. essentially, these are our goals (e.g., describe; under-
Similarly, data analyses and the vast array of statisti- stand; explain; intervene when needed, possible,

4
Methodology: What It Is and Why It Is so Important
Copyright American Psychological Association

and desirable) and these are our means (use of theory, enced in part by ovulation cycles (e.g., Haselton
methodology, guiding concepts, replication of & Gildersleeve, 2011).
results). Science is hardly a game because so many ■■ When prisoners come before a parole board, are
of its tasks and topics are so serious—indeed, a there any unexpected influences on the deci-
matter of life and death (e.g., suicide, risky behavior, sion of whether they can be released before
cigarette smoking). Yet there are rules and there are their prison sentence is complete? Surprisingly,
enormous benefits to be gained by all sciences and the point during the day at which a given pris-
scientists. Think of the chaos if methods varied oner sees the parole board is relevant to the
across countries or professions; we simply could not outcome. An evaluation of multiple parole
accumulate an agreed-on body of knowledge. decisions revealed that the likelihood of being
Second, methodology is needed to identify, granted parole is much higher in the morning
detect, isolate, and reveal many of the extremely and immediately after a lunch break than at
complex relations that exist in the world. Science other times (Danziger, Levav, & Avnaim-Pesso,
uses special controlled arrangements and special 2011). Indeed, as hunger (or fatigue) increases
methods (e.g., equipment, measures) to isolate and as lunch time approaches, the chances of
influences that are otherwise difficult, if not impos- being paroled decrease, but they bounce up again
sible, to detect from casual observation in everyday right after the lunch break. The same raters were
life. Consider a brief sample of findings from the involved, and the result cannot be explained by
natural and social sciences conveying the complexi- severity of the crimes or types of prisoners.
ties of our world that the methods of science were ■■ Do early harsh environments for children (e.g.,
needed to reveal. Consider the guiding question in exposure to violence, enduring stress, corporal
the examples and the answers that scientific method punishment) have any long-term effects? Yes,
provided: they can lead to many untoward outcomes,
including poor academic performance (e.g., poor
■■ What is near the boundary of our universe? Well,
grades, dropping out of school) and mental ill-
for starters, a galaxy (a system of millions of stars
ness (e.g., posttraumatic stress disorder, depres-
held by gravitational attraction) has been identi-
sion, anxiety). Also, the outcomes can include
fied that is more than 13 billion light-years away
enduring impairment of the immune system
(e.g., Maartens, 2013).
(ability to ward off infection and inflammation)
■■ How did dinosaurs become extinct?
and are likely the reason why many such chil-
Approximately 66 million years ago (give or take
dren have premature deaths from serious disease
300,000 years), a huge asteroid (15 kilometers,
much later in adulthood (e.g., Krug, Dahlberg,
or more than 16,400 yards, wide) crashed into
Mercy, Zwi, & Lozano, 2002).
the earth (near Yucatan, Mexico) and led to the
extinction of more than half of all species on the The findings in these examples required very
planet, including the dinosaurs. The material special observation procedures under special
blasted into the atmosphere led to a chain of arrangements, measures, assessments, and methods
events that resulted in a global winter (e.g., of data evaluation. The conclusions I list are not
Brusatte et al., 2014). discernible by everyday observation. If you said, “I
■■ Are male and female interactions and behaviors knew all along based on my casual observations that
influenced by a woman’s menstrual cycle? Where there was a galaxy at the boundaries of our universe;
a woman is in her menstrual cycle apparently what’s the big deal?” or “Of course prisoners who
has an effect on her behavior (e.g., selection of are seen after the parole board’s lunch break are
clothing, gait when walking, and the type of man more likely to be granted parole,” you are among
that seems attractive) and how men respond to a very elite group. The rest of us needed careful
it. All of this occurs outside of consciousness but research and scientific methods to grasp these
conveys dynamically changing interactions influ- phenomena!

5
Alan E. Kazdin Copyright American Psychological Association

Third, whether the relations are complex or not, methodological practices are designed to help man-
for many questions of interest extensive information age and overcome them.
(a lot of data) are needed to draw conclusions. How
to obtain that information (assessment, sampling) Brief Illustrations of Our Limitations
requires very special procedures to yield trustworthy in Accruing Knowledge
results. For example, how many individuals experi- Senses and their limits. The limitations of
ence some form of psychiatric disorder? To answer our senses—including vision, hearing, and smell—
this question, one needs a large sample, a represen- serve as a familiar example to convey how very
tative sample, and special procedures (e.g., use of selective we are in the facets of reality that we
measures known to provide consistent information can detect. We consider what we see, hear, and
and to reflect the phenomenon of interest). As it smell to represent reality, that is, how things are.
turns out, approximately 25% of the U.S. popula- But this reality is very selective. For example, we
tion at a given point in time meet criteria for one or
see only a small portion of the electromagnetic
more psychiatric disorders (Kessler & Wang, 2008).
spectrum and refer to that as the visible spectrum.
Approximately 50% experience a disorder at some
Probably a better term would be the human visual
point in their lifetime. This kind of information
spectrum. We cannot see infrared, or ultraviolet,
cannot be obtained from casual observation or indi-
for example. Other animals (e.g., birds, bees and
vidual experience. Large data sets and systematically
many other insects) see part of the spectrum we do
collected data are needed to address many ques-
not see, which helps with their adaptation (e.g.,
tions, and science is needed to provide the informa-
identifying sex-dependent markings of potential
tion in a trustworthy, consistent, transparent, and
mates that are only visible in ultraviolet light).
replicable way.
The same is true for sounds and smells; many
Finally, we need science to help surmount the
nonhuman animals have senses that evaluate dif-
limitations of our usual ways of perceiving the
ferent parts of the world from those we can expe-
environment and reaching conclusions. Along with
these limitations in our perceptions, there are many rience. Many animals can hear sounds that we do
sources of subjectivity and bias that interfere with not hear (e.g., dogs, elephants, pigeons) and have
obtaining more objective knowledge—that is, infor- a sensitivity to smell that vastly exceeds our own
mation that is as free as possible from subjectivity (e.g., bears, sharks, moths, bees). More generally,
and bias. How we perceive and think is wonderfully many nonhuman animals trump our vision, hear-
adaptive for handling everyday life and the enor- ing, and smell or have differences that are not
mous challenges presented to us (e.g., staying out better (more sensitive) or worse but just different
of danger, finding mates and partners, rearing chil- (e.g., seeing different parts of the electromagnetic
dren, adapting to harsh and changing environments, spectrum).
meeting the biological needs of ourselves and our These examples are intended to make one point:
family—it is endless). Evolution spanning millions As humans, we see one part of the world, and that
of years has sculpted, carved, sanded, and refined picture is quite selective. The picture we have of
these skills. Yet those very adaptive features can what is omits piles of things that are. So one rea-
actually interfere, limit, and distort the information son for science is to overcome some of the physical
presented to us and do so by omission (our percep- limitations of our normal processing of informa-
tion omits many facets of experience that we do not tion. Much of what we want to know about and see
detect well) and by commission (we actively distort cannot be discerned with our ordinary capacities
information on a routine basis). Scientific methodol- (our senses). In fact, much of what we have learned
ogy has emerged in part to surmount the limitations about the universe and also about interpersonal
of more casual observation. interaction and attraction comes from what is not
That said, a few limitations are worth noting. obvious, detected, or detectable by means of usual
Science does not get rid of these limitations. Rather, sensory perception.

6
Methodology: What It Is and Why It Is so Important
Copyright American Psychological Association

Cognitive heuristics. Leaving aside physical & Johnson, 2006). Objective facts about the material
limitations in seeing, smelling, and hearing the presented can be carefully controlled in research
world, more persuasive arguments for the need to allow demonstration of ethnic biases in how
for science come from many areas of cognitive participants react to stereotypes and biases they
psychology. These arguments are more persuasive would not otherwise express. More generally, if
in the sense that when we look at experience well we believe that one ethnic group behaves in this or
within the capacities of our senses, we may still that way or that people from one country or region
have enormous limitations in how we process have a particular characteristic, we will see evidence
that information. You already know the everyday that supports it—the supportive evidence is more
expression “seeing is believing”; psychological salient in our mind and memory and is constructed
research has provided considerable support for rather than recording the incoming data objectively.
the additional claim “believing is seeing.” We Counterevidence does not register as salient or, if
process the world in special ways, and various and when it does, is dismissed as an exception.
cognitive processes have been well studied. These Cognitive heuristics are not the only set of influ-
processes can and often do systematically distort ences that guide our perception. Our motivation and
and lead us to make claims and inferences that mood states can directly influence how and what
do not reflect reality, as revealed by less biased or we perceive of reality (Dunning & Balcetis, 2013).
unbiased means. Both biological states (e.g., hunger, thirst) and psy-
Several characteristics of normal human func- chological states (e.g., mood) can directly guide
tioning, referred to as cognitive heuristics, reflect how reality is perceived. This is sometimes referred
how we organize and process information. These to as motivated perception or wishful perceiving.
processes are out of our awareness and serve as For example, when a person feels threatened or
mental shortcuts or guides to help us negotiate angry, he or she is likely to see another as holding
many aspects of everyday experience (Kahneman, a weapon rather than a neutral object (Baumann &
2011; Pohl, 2012). These guides help us categorize, DeSteno, 2010). That is, the reality we perceive is
make decisions, and solve problems. The heuristics influenced by us as a filter, and our changing bio-
emerge as bias when we attempt to draw accurate logical and psychological states have an impact on
relations based only on our own thoughts, impres- what we see, hear, and recall. Obviously, motivated
sions, and experience. There are several cognitive perception can have life-and-death consequences
heuristics, but let me convey a sample to make con- because the person perceiving (e.g., civilian, police
crete what I am talking about. officer) feels threatened and acts accordingly. We
The confirmatory bias reflects the role of our are not likely to be empathic when we hear a per-
preconceptions or beliefs and how they influence son shot someone else when in fact there was no
the facets of reality we see, grasp, and identify. danger. The “in fact” may not have been so relevant
Specifically, we select, seek out, and remember
because the perception of the individual who fired
evidence in the world that is consistent with and
was guided by perceived threat. My comments are
supports our view. That is, we do not consider and
not about blame or justification; rather, they are
weigh all experience or the extent to which some
intended to convey that reality is filtered and that
things are or are not true on the basis of the reali-
filter can be biased and influenced in ways quite dif-
ties we encounter. Rather, we unwittingly pluck out
ferent from the actual facts or events.
features of reality that support (confirm) our view.
This is particularly pernicious in stereotypes, as one Memory. Other examples illustrate how our
case in point. For example, experimental manipula- normal processing of information influences and
tion of ethnic characteristics (e.g., skin tone among distorts and, again, why we need assistance from
African Americans, ethnicity of victims in a crime) methodology to help surmount these influences.
leads to different evaluations of crime and sentencing Memory refers to the ability to recall information
practices (e.g., Eberhardt, Davies, Purdie-Vaughns, and events, although there are different kinds of

7
Alan E. Kazdin Copyright American Psychological Association

memory and different ways of studying them. As can we induce them in stories, memory tasks, and
humans we believe (and are often confident) that laboratory studies (e.g., Brainerd & Reyna, 2005)?
our memory records reality, but research has very Yes, experiments have shown that they can even be
clearly shown that we recode reality (Roediger & implanted. In these experiments, when people were
McDermott, 2000). That is, more often than not asked to recall material, false memories (things that
we do not recall things as they happened. And did not occur at all) were often recalled and mixed
this has come up in many contexts. with those that had occurred. The key is that people
First, as we consider stories of our past (e.g., do not see them as false memories, nor do they flag
childhood, high school years), little details and some memories as accurate or true and others as
sometimes larger ones get filled in and become part implanted. When someone says he or she remem-
of our remembered story. Our memories draw on bers something perfectly or well, it may be useful to
information for experience of the external world, regard that as a statement of confidence in a memory
but these memories are filled in by internal pro- rather than in accuracy of the account.
cesses (e.g., imagination, thought). As we recount Finally, consider recall, used heavily by the
a story, we cannot distinguish between what in the courts in legal proceedings. In jury trials, the most
story actually happened and what did not. persuasive type of evidence is eyewitness testimony.
Reality monitoring is the name for the memory Juries are persuaded by a witness on the stand say-
function that differentiates memories that are based ing he or she saw the defendant do this or that and
on external (the world) versus internal (one’s own perhaps even identified the defendant in a lineup
thoughts, perceptions; Johnson, 2006). Thus, I can as the perpetrator. The reliance on eyewitness tes-
separate my imagined phone call from the Nobel timony makes forensic psychologists want to jump
committee (last night’s dream) from reality (the out of their basement windows because rather
phone call I actually received yesterday was from extensive research has shown that this type of tes-
my dry cleaner—I had to pick up my shirts, or they timony is the least reliable form of evidence and is
would be thrown out). Errors occur when that dis- responsible for sending more innocent victims to
tinction is not made, which is a function of several prison than any other form of evidence (Wells &
things, including how vivid the imagined events Loftus, 2013). Clearly, memory, perception, and
are and how consistent they are with the external confident accounts must give one pause or caution.
stimuli. We develop a story or scheme of an event General comments. Several facets of percep-
or occurrence and fill in details where needed, and tion, thoughts, and emotions influence how we
when we recall the event, we cannot always distin- characterize the world, although I have men-
guish the source. Sometimes our own mind fills in tioned only a small sample (e.g., only one cogni-
details, and sometimes this process is aided by the tive heuristic, although there are several; only a
stories others have told us that become our stories few areas of memory research, including reality
and are planted as part of our experience. monitoring, false memories, and eyewitness tes-
Second and related, the notion of false memories timony, while omitting others). The point was
has appeared in the public as well as the scientific just to convey that, as humans, we have limita-
literature. Interest in false memories emerged from tions that can readily influence the conclusions
the experiences of many clients in therapy who, we reach. These limitations can have little impact
over the course of treatment, newly recalled child- (e.g., details regarding who was at a social event
hood experience of abuse. In several cases, it in last month and who drank and ate what) or enor-
fact appears as though the memories were actually mous impact (e.g., who goes to jail or receives the
induced by the very process of therapy. This does death penalty). Also, humans negotiate life rather
not mean, of course, that all, most, or any given rec- well. As a rule, we do not bump into buildings
ollection of abuse is false, but we know that some or each other when walking down the street, put
are, and that is just enough to establish that it can on our clothing correctly most days, and say “hi”
happen. Researchers have studied false memories— rather than “goodbye” when we first encounter a

8
Methodology: What It Is and Why It Is so Important
Copyright American Psychological Association

friend or colleague during the day. So we should so the full range of human characteristics (e.g., com-
not distrust our senses, cognition, and affect or be mitment, integrity, and creativity and also decep-
dismissive in any way about their utility. tion, fraud, and falsely claiming credit) is present.
Accumulating scientific knowledge is another Methodology is the best we have now in the way of
story. The limitations I have illustrated convey how developing the knowledge base. Methods are con-
essential it is to develop the means to counter nor- stantly evolving to improve what we know and how
mal experience, perception, memory, and the like we know it and to correct sources of bias or influ-
in developing a knowledge base of our world. The ences that can interfere with obtaining knowledge.
challenge is as follows: We know we have limita-
tions in our perception and hence in our ability to
KEY TENETS AND STRATEGIES
acquire unbiased knowledge without some system-
atic set of aids. We need more reliable tools to codify A few overarching tenets or principles guide science
current and past experience and surmount some of and the methods we use to obtain knowledge. These
our normal recall and other limitations. The para- are useful at a general level to understand science.
dox is this: We ourselves, with these imperfections, In addition, they are extremely useful at a very con-
have the responsibility for developing the tools crete and specific level, namely, in interpreting the
(methods) to surmount those limitations. results of a given study and in communicating the
Think of science as a way of knowing filled with results of a study to one’s colleagues. In this section,
checks and balances. For example, one check, argu- I describe translating ideas into testable hypotheses,
ably the most important, is repetition of findings parsimony, plausible rival hypotheses, replication,
by other investigators. This repetition of findings and caution and precision of thinking as core
is referred to as replication. For example, if I find elements.
an amazing result and no other investigator can First, ideas for scientific research must be trans-
reproduce (replicate) that result after many excel- lated into testable hypotheses. Scientific research
lent tries, my finding is suspect. I am not necessarily depends on putting ideas to a test, which means
suspected of anything odd, but the finding is not making predictions, using systematic measures, and
reliable. Perhaps the finding depended on something evaluating whether the data do or do not support
no one knows about or occurred by chance, was a a hypothesis. The concept of falsifiability has been
fluke, or happened because of a bias I did not detect used as part of the notion of testing ideas. The idea
or control. At this moment in our discussion, the must be one that can be put to a test and in principle
reason does not matter (although all of this will be shown to be false.
discussed later), but we have to say that my finding In everyday life, one can see that this require-
is not to be taken as a reliable finding, and we go on. ment is rarely invoked (or needed). For example,
Perhaps some people replicate my finding, but others we might say a person is passive–aggressive. Usually
do not. This suggests that some other condition that means we are interpreting their behavior as
or circumstance (e.g., some characteristic of the par- being nasty even though it does not appear that way.
ticipants, how the experimental manipulation was In everyday life, the concept conveys a point, and
conducted) may influence whether the finding is we usually do not challenge the person making it.
obtained. These possibilities can be readily studied. To translate the concept into a scientific hypothesis
If my study cannot be replicated, that is annoying at (not too difficult to do), we would need a measure
the moment, but we are committed to the process, (systematic, objective, reliable, valid) of passive-
and the last thing we want is false knowledge—that aggressive style or behavior and to specify what
is, findings that do not hold up across investigators, evidence from our study would support or refute
laboratories, and time. the view that passive–aggressiveness could explain
Methodology does not eliminate bias and prob- the behaviors of interest. Perhaps the person is busy,
lems, and so a great dose of humility about the pro- slow, not wildly competent, or forgetful. We need a
cess is just wise. Also, science is a human enterprise, concrete way to test (and possibly support or refute)

9
Alan E. Kazdin Copyright American Psychological Association

that passive–aggressiveness, rather than these other orbited around the sun (heliocentric view) rather
constructs, explains the behavior. than around the earth (geocentric view). This lat-
Occasionally, within psychology, theories are ter geocentric view had been advanced by Claudius
advanced that include components that are not easily Ptolemy (ca. 85–165), a Greek astronomer and
testable. For example, psychoanalysis, developed mathematician. Ptolemy’s view had dominated for
by Sigmund Freud (1856–1939), was a comprehen- hundreds of years. The superiority of Copernicus’s
sive theory proposing that psychological processes view was not determined by public opinion sur-
and specific early childhood experiences play criti- veys or the fact that Ptolemy was no longer alive to
cal roles in our everyday behavior (e.g., comments defend his position. Rather, his account could bet-
we make to others, relationships), psychological ter explain the orbits of the planets as well as other
defenses to ward off threatening material (e.g., phenomena and do so more simply, that is, parsimo-
denial, projection), and psychiatric disorders or niously. The heliocentric view could explain more
social, emotional, and behavioral problems (e.g., with one key construct.
anxiety, depression, aggression) and in how to Parsimony relates to methodology in concrete
change maladaptive functioning in the context of ways. When an investigation is completed, we ask
psychotherapy. Among the criticisms is that the the- how to explain the findings or lack of findings.
ory is not very testable. Many key components could The investigator may have all sorts of explanations
not be shown to be false because the theory seemed for why the results came out the way they did.
to be slippery enough to explain results no matter Methodology has a whole set of explanations that
what happened. A key facet of science is devising may be as or more parsimonious than the one the
testable hypotheses. Within the tradition of contem- investigator promotes. For example, say I develop a
porary science, the concept of falsifiability, that is, new psychotherapy (Kazdin’s mindlessness therapy)
being able to show that an idea or hypothesis is in and show that it is better than no treatment. I now
fact not true, is critical. In short, a key component of explain how engaging in mindless behaviors (e.g.,
science and the methodology on which it draws is to wandering the streets, grocery shopping, counting
translate ideas into testable hypotheses and hypoth- backward from 100 in Sanskrit) leads to reduced
eses that can be demonstrated to be false or that can depression. I might be right. Yet, my view is not par-
be supported. simonious and ought not to be adopted yet. There is
Second, scientific knowledge is based on parsi- a large literature showing that doing anything (e.g.,
mony. Parsimony refers to the practice of providing meeting with a therapist, attending sessions) and
the simplest version or account of the data among expecting improvement in treatment leads to thera-
the alternatives that are available. This does not in peutic change. These latter influences are referred
any way mean that the explanations themselves are to as common factors of therapy because they are
simple. Rather, it refers to the practice of not adding present in many techniques. Common factors are
all sorts of complex constructs, views, relationships more parsimonious than my mindless interpretation
among variables, and explanations if an equally because common factors already explain the findings
plausible account can be provided that is simpler. from hundreds of treatment studies. We do not need
We add complexity to our explanations as needed. If another set of constructs to explain the findings from
two or more competing views explain why individu- my study. Additional research is needed to show
als behave in a particular way, we adopt the simpler that we need my explanation, but on the basis of
of the two until the more complex one is shown to my study and its design (just a no-treatment control
be superior in some way. group), there is no need for that explanation now.
A well-known illustration of competing inter- Third, plausible rival hypothesis is another key
pretations comes from cosmology and pertains to concept of science (Cook & Campbell, 1979). A
the orbiting of planets in our solar system. Nicolas plausible rival hypothesis refers to an interpretation
Copernicus, a Polish scientist and astronomer of the results of an investigation that is based on
(1473–1543), advocated the view that the planets some other influence than the one the investigator

10
Methodology: What It Is and Why It Is so Important
Copyright American Psychological Association

has studied or wants to discuss. The question to ask are all plausible, and one cannot decide from the
at the completion of a study is whether other plau- study whether cough drops make any difference.
sible interpretations exist to explain the findings. Methodology is all about plausible rival hypoth-
This sounds so much like parsimony that the distinc- eses and designing studies so that some hypotheses
tion is worth making explicit. Parsimony refers to or interpretations are made more plausible than
adopting the simpler of two or more explanations others. We engage in various methodological prac-
that account equally well for the data. The concept tices (e.g., random assignment, use of comparison
is quite useful in reducing the number and complex- and control groups, keeping experimenters naïve
ity of concepts that are added to explain a particular with regard to the hypotheses) to make some inter-
finding. Plausible rival hypothesis as a concept is pretations implausible so that the interpretation
related to parsimony but has a slightly different we are evaluating can be better evaluated. In the
thrust: At the end of the investigation, can other example, we might control for some of the interpreta-
plausible interpretations be made of the finding than tions I added to give a better test of the cough drop
that advanced by the investigator? Simplicity of the hypothesis.
interpretation (parsimony) may or may not be rel- Fourth, as mentioned previously, replication is
evant. At the end of the study, there could be two or central to science. Replication refers to repetition
10 equally complex interpretations of the results, so or repeatability and is important for two reasons.
parsimony is not the issue. To begin, the procedures used in research must
For example, let us say we have an amazing be repeatable. If one were to ask another scientist,
hypothesis that people who take cough drops dur- “How did you get that finding?” the unacceptable
ing a cold get better faster than those who do not. and inappropriate answers (drawn from childhood)
We do a massive survey of students and identify two are “I’m not telling!” or “That’s for me to know
groups—those who take cough drops when ill and and you to find out.” Science operates so that what
those who do not. We keep track of the participants investigators do in a study, how they do it, and all
and closely monitor all those who get colds in the of the circumstances are described. Others must
next few months. Then we call them every day and be able to repeat the study. One might refer to this
find out when their cold is over. Lo and behold, we as replicability of the procedures. The second way
find that those individuals who take cough drops get in which replication is central regards whether the
better in one half of the time it takes those who did findings can be reproduced. A question about the
not take cough drops. We are all ready to conclude results of any study is whether the findings can be
that, as predicted, taking cough drops is helpful in repeated or obtained again by someone using identi-
limiting the duration of colds. cal or similar procedures.
Plausible rival hypotheses come in here by rais- Replication relates to parsimony and plausible
ing the question “Are there other plausible explana- rival hypotheses. Were the results evident in this
tions for the results?” The answer is a resounding one study because of a chance effect or some odd
yes. It may be that people in the cough-drop group circumstance in the situation about which the inves-
are just healthier in general. Perhaps those who take tigator may be unaware? Could there be a simple
cough drops tend to take better care of themselves (parsimonious) explanation or one that is equally
in general (better eating habits, more exercise), take plausible (rival hypothesis)? Replication is needed
care of themselves during a cold more (a little more to establish the credibility and genuineness of the
bed rest and chicken soup), or have a history that finding. It would be unthinkable for a researcher to
indicates they are healthier to begin with. Perhaps say he or she demonstrated a cure for a type of can-
they would have colds of shorter duration for these cer but that other scientists could not replicate the
reasons, and cough drops are not needed at all. We results. One study might excite the news media and
could provide many more explanations for these the public who reads the story. Yet, the scientific
findings. They do not have to be more or less parsi- community might be very skeptical until this finding
monious than the cough drop interpretation. They is replicated by other investigators. Only through

11
Alan E. Kazdin Copyright American Psychological Association

repeated demonstration does one gain confidence younger due to this cosmetic cream, and will that
in a finding. The initial investigation that obtained special new diet really cause weight loss? The ques-
the result is tantalizing, promising, and maybe true, tions are simplistic because they suggest that one
but more is needed. Skepticism of a finding is not variable leads to a particular outcome and that the
suspicion directed toward the original investiga- relation is causal. Information in daily life moves
tor; rather, it comes from the realization that many seamlessly from casual to causal inferences. Science
influences might explain the results. For example, a encourages greater care.
finding might emerge because of chance, a statistical For example, we know that having a cigarette
artifact that is embedded in the procedures. That is, smoker in the home increases the likelihood that an
sometimes the data will show a statistically signifi- infant will die from sudden infant death syndrome
cant effect even if there is no real effect in the world. (SIDS Network, 2015; sudden infant death refers
Finally, science encourages caution and precision to the death of an infant, usually between the ages
in thinking. This does not mean scientists (humans) of 2 weeks and 2 years, that cannot be traced to
are invariably cautious and precise. Rather, the sci- disease, physical abuse, or other disorders. Many
entific community to which we belong demands of more children die of SIDS in a year than die of can-
us as investigators that we are careful in not going cer, heart disease, pneumonia, child abuse, AIDS,
beyond the data, or at least not too far. This means cystic fibrosis, and muscular dystrophy combined).
we can describe only what the study demonstrated, Hearing the relationship I mentioned (smoking as
and any other part remains to be determined by a risk factor) in casual thinking almost naturally
further research. For example, in research we distin- leads one to assume smoking is the culprit (cause).
guish key concepts such as correlation, risk factor, Moreover, it is even more tempting to move to
and cause. A correlation indicates that two vari- interventions—we ought to stop smoking to decrease
ables are related to each other at a particular point SIDS. Scientific thinking is a bit more cautious.
in time. A risk factor indicates that two variables Smoking as a cause has not been established. Could
are correlated with each other but that one clearly it be one of many causes? Could it perhaps not be a
comes before the other. An example of a risk factor cause at all but be correlated with something that is
is cigarette smoking, which is a risk factor for later more likely to play a causal role? These questions and
lung and heart disease. That is, those who smoke are of course the very process of questioning is science.
more likely to contract later disease. We know the The tenets and practices I have highlighted pro-
ordering of these. A cause means that one variable vide the underpinnings of methodology and are
leads to, produces, and is responsible for a particular points to which we as scientists return in our think-
outcome. For example, nonhuman animal research ing about a particular study and its findings and
has shown that various experiences early in develop- conclusions. Healthy skepticism about a given find-
ment (e.g., physical contact with a parent, ingestion ing or set of findings is also part of methodological
of toxins such as lead) influence brain development. thinking. This skepticism is reflected in thoughts
The studies can show a causal relation because pre- about a study and includes what might be called
senting or withholding experiences dictates the out- “yes–but” questions. Scientists read a study or learn
come. When a study shows only a correlation, the of a finding and say, “Yes, but . . .”
scientific community and the investigators them-
selves are cautioned not to go beyond the data and ■■ are there simpler interpretations to explain the
assume or state something more. findings than what the author has concluded
The caution and precision of scientific statements (parsimony)?
and inferences stand in contrast to inferences drawn ■■ is there is a more reasonable interpretation
in everyday life. The public at large and the media than what the author concluded (plausible rival
freely use such terms as because, due to, and cause hypothesis)?
in connecting concepts; for example, are children ■■ does the relation (correlation, risk factor) really
aggressive because they were abused, will I appear explain the connection of these variables?

12
Methodology: What It Is and Why It Is so Important
Copyright American Psychological Association

■■ has that finding been replicated? many of the main sources of bias and artifacts that
■■ do the design, assessments, and data analyses allow can interfere with drawing valid inferences. There
for the conclusions to be reached, whatever their are others referred to in earlier citations, but this set
interpretation (caution and precision in thinking)? is a great place to begin.
Some of the problems in research are so perva-
Skepticism does not mean rejection of findings or
sive that one can almost always point them out. For
criticism for its own sake. We build the knowl-
example, we have learned from years of reviews
edge base, often slowly, by clarifying, elaborating,
from the 1960s to the present that most studies in
repeating, and better establishing what we think we
psychology do not provide very good tests of their
know. Skepticism in asking the “yes–but” questions
hypotheses (e.g., Bakker, van Dijk, & Wicherts,
operates at its best when investigators adopt this
2012; Cohen, 1962). The key reason is that they
stance about a finding rather than their peers.
have low statistical power—too low to find a dif-
ference even if there really is a difference. Power
DRAWING VALID INFERENCES
depends on many factors in a study, but sample size
The overall purpose of methodology is to permit is the first place to look. Studies typically use too
one to draw sound or valid inferences, that is, to few participants to find real differences.
reach conclusions that are as free as possible from But if this problem is not new, why is it still with
competing interpretations (plausible rival hypoth- us? That is a separate topic, but perhaps the best
eses). We engage in the practices and procedures of answer is that psychology as a discipline does not
methodology not for their own sake but to help with enforce or require well-powered studies. Also, we
interpretation and to bring clarity or relative clarity have learned that repeatedly pointing out the prob-
to our findings. In this process, methodology has lem and pleading with researchers to increase power
two major roles. does not have very much impact (see Sedlmeier &
Gigerenzer, 1989). In general, it is wise to know on
Codifying Sources of Problems the basis of such issues as statistical power whether
Methodology codifies the sources of problems that one could obtain the desired effect well in advance
emerge in drawing inferences. Essentially, meth- of running the first participant.
odology provides a list of most of the problems to I mentioned that methodology codifies many of
be wary of, what the investigator ought to think the problems that can emerge in conducting and
about before running the first participant, and what interpreting research, and low power as a result of
might go wrong during a study that would interfere small sample size is just one of them. (Later chapters
with drawing clear conclusions. In everyday life, will discuss power and other methodological prob-
we develop fears and worries based on experience; lems as well.) The overall point, for our purposes in
some biological predisposition; information from this introductory chapter, is that methodology pro-
relatives, peers, and the media; and no doubt many vides guidelines—fairly clear guidelines, actually—
other sources. We have some friends who worry about where to look for potential problems that
about exams, elevators, catching a disease, being could undermine a study. Most of these problems
struck by lightning, and so on—it seems as though can be identified and addressed at the design stage
it would be so much better if there were a master list before a study is begun.
of worries somewhere from which we could select!
Well, in the context of science, methodology pro- Codifying Solutions
vides a master list of sources of worry or concern. Methodology also codifies many of the solutions to
What to worry about is not a very sophisticated the problems and practices that can help scientists
term. Understandably, other terms used in meth- draw valid inferences. Here, too, it would be useful
odology include sources of artifact and bias and to have a list of solutions to the problems we worry
threats to experimental validity (e.g., Kazdin, 2016; about in everyday life and outside of the context of
Shadish, Cook, & Campbell, 2002). Table 1.1 lists methodology. How great would it be to, on the back

13
Alan E. Kazdin Copyright American Psychological Association

TABLE 1.1

Examples of the Problems in Drawing Valid Inferences and Some Practices to Address Them

Name of the problem Definition Possible solutions


History and maturation Changes that occur over time as a result of Include a control group that does not receive the experi-
events (history) or processes within the mental manipulation or intervention. If the groups are
individual (maturation). The results of composed by random assignment, the historical and
the study may perhaps be due to such maturational influences will be controlled. It is likely
changes rather than to the manipulation, that such influences will apply to both (all) groups,
intervention, or independent variable. and group differences are likely to be due to the
experimental manipulation.
Testing Taking any test on more than one occa- A control group that receives the same assessments
sion often leads to changes in perfor- but does not receive the intervention or experimental
mance (e.g., improvement). Studies manipulation makes repeated testing implausible as
that assess participants on multiple an explanation for any group differences.
occasions (e.g., pretest and posttest)
might show change just because of
repeated testing.
Selection biases The groups (e.g., experimental, control) Assign participants randomly to conditions. With
are different to begin with because of reasonably large samples, this random assignment is
how they were selected or formed. likely to produce groups that are equivalent before the
Any differences between groups at the manipulation is provided. Alternatively, participants
end of the study may be due to these can be matched on variables (e.g., level of anxiety)
differences rather than to anything the and randomly assigned to groups in matched sets so
investigator does. groups are equivalent on key variables.
Attrition (dropping out) Loss of participants over the course of the Try to minimize the loss of participants. Also, evalu-
study can make groups different. The ate characteristics of participants who did drop
random composition of the groups has out versus those who did not. Complete statistical
changed because participants selectively tests designed to control for the impact of loss of
excluded themselves. The groups may participants (e.g., intention-to-treat analyses or other
be different from each other, leading to a methods of imputation [data estimation]).
selection bias, as noted previously.
Cues of the experimental Incidental cues of the experiment (what Use a control group that receives all but the special part
situation participants believe, what they are told, of the intervention so that almost all cues are identi-
the expectations unwittingly conveyed cal across groups. Interview participants in the study
about how they ought to perform) may or in pilot work to ask how they are likely to behave
explain the group differences rather to determine whether cues would lead to a systematic
than the experimental manipulation. way of responding. Use measures that are not so
Cues may foster a way of responding transparent that participants can readily discern what
that account for the results. is being measured.
Sample characteristics The findings may be restricted to the spe- Include different types of participants (e.g., males,
cial sample that was used; the finding females; more than one ethnic group). College student
is genuine but may not apply to others samples, used in most psychological research, have
(e.g., of different ages, sex, ethnicity, special characteristics that may make them unique.
culture). Think carefully about the sample that provides the
best test of the hypotheses. Analyze the results in a
way that permits one to see whether a characteristic
of the sample in fact relates to the finding; replicate
results with other samples.
Sample size (low power) No differences were obtained in the study Use a larger sample. Use a within-subjects design
because the power (ability to detect a (pre and post measures on the same participants).
true difference when there is one) was Also, make directional statistical predictions and use
too low. directional (one-tailed) statistical tests. Use statistics
(such as effect size) other than those that focus on
statistical significance.

14
Methodology: What It Is and Why It Is so Important
Copyright American Psychological Association

TABLE 1.1 (cont.)

Name of the problem Definition Possible solutions


Questionable reliability No statistically significant differences Use measures that have validity and reliability data per-
and validity of the were obtained because the measure tinent to the focus of the investigation. Use multiple
measure has considerable error variability measures and combine them statistically (e.g., by
(unreliability) or it is not clear that this factor analysis) for a better index of the construct of
measure is a very good measure of the interest. Use multiple measures that rely on differ-
construct of interest. If a researcher ent methods (e.g., self-report, direct observation).
invented a measure and the results Conduct analyses on the measures to assess directly
did not come out as expected, it could within the study whether measurement issues might
easily be the case that the items (face explain the findings (e.g., internal consistency, test–
validity) do not measure what you retest reliability). If you invent a measure, include
believe or do not measure it very well some data to suggest that it is valid as an index of the
(face validity). concept you wanted to measure.

Note. The purpose of this table is to illustrate how methodology codifies problems to which researchers must be
sensitive in their demonstrations and solutions to rule out or address these problems. The list of problems or solutions
presented here is not exhaustive (see Cook & Campbell, 1979; Kazdin, 2016; Shadish, Cook, & Campbell, 2002).

of the sheet that says worry about lightning, Ebola depression. Also, if I include only college students
virus, mad cow disease, and germs from friends, who seem depressed on a self-report measure, this
unprotected sex, and unprotected celibacy, have a may limit the extent to which the results might apply
long list of solutions? In the context of research, to patients with clinical depression.
methodology provides a long list of solutions and Also, methodological challenges and solutions
strategies. In Table 1.1, I have listed in the third are evolving. Ethical issues, for example, constantly
column some of the solutions that methodology require new considerations and guidelines. Will
encourages. These are only a sample of some of the using social media as part of my study create new
major solutions to some of the major problems. One risks for violating confidentiality? Is the database
can see that we engage in research methodological protected (encrypted) so that no one can invade the
practices (solutions) to address specific problems. computer storage and uncover confidential informa-
If methodology were merely a list of problems tion and the identity of the participants associated
and solutions, however, then one might not need a with that information? When my study is com-
course or a book. One could just master the lists and pleted, I may make the raw data and analyses freely
go on to other topics. The challenge is that design- available to other scientists (as many journals now
ing a study involves many decision points, and each require). Are there any risks with that, for example,
of these points has some implications for drawing might the data be used in a way that unwittingly
conclusions. Consider that I want to study new treat- portrays a particular population (e.g., ethnic group,
ment for depression. Okay, who will be the partici- workers in a particular industry) or geographical
pants, and how will I select them? The answer to this area (e.g., region of United States, country) in a
question can determine whether treatment works negative light? Research methods evolve (e.g., com-
and, if it does work, whether this effect can be shown bining large data sets on the basis of how individuals
statistically. For example, if I recruit individuals of use the Web or their electronic health records) and,
any age who are depressed (e.g., 15–60 years) and as they evolve, how individuals are protected also
individuals who are depressed for whatever reason needs to change.
(e.g., clinical depression, bereavement), I may be Research situations require consideration of what
less likely to find an effect because of the variability problems will emerge and what solutions are possi-
or individual differences in participants and types of ble and the trade-offs of one solution versus another.

15
Alan E. Kazdin Copyright American Psychological Association

As in many other aspects of life, sometimes choosing area has a need for basic research on processes,
one solution limits other courses of action. That is, mechanisms of action, and so on and eventually, one
maximizing one option may lead to sacrifice of some hopes, intervention trials (treatment, prevention).
other benefit. Methodology requires a deeper under- Progress is slow even when all the methodological
standing of practices so that these trade-offs can be stars are aligned (i.e., strong empirical tests based
thoughtfully considered in relation to the specific on great methods). Among the challenges that make
hypothesis the investigator wants to test. the progress slow is replication of effects and then,
if relevant to the public at large (e.g., clinical prac-
General Comments tice, child rearing, day care), dissemination of the
How methodology codifies various problems and results. This is not a quick process under the best of
solutions that relate to drawing valid (well-based) circumstances. Methodologically weak studies along
inferences might sound too abstract and cerebral to the way merely delay or even mislead us as we move
be of broad relevance beyond the confines of scien- forward.
tists conducting studies. Actually, issues can affect
decisions in everyday life and indeed even deci-
METHODOLOGY IN CLINICAL
sions about life and death. For example, we are all
PSYCHOLOGY
keenly interested in the development of treatments
for life-threatening illnesses. How could methodol- Although methodology is central to all science in
ogy figure into this? A review of medical research general, there are special reasons to focus on issues
of a variety of diseases and conditions revealed that related to research in clinical psychology. Clinical
more than 25% of the studies surveyed (published psychology embraces all of the usual features of sci-
from 1975 to 1990) revealed no differences among entific research, such as defining the research idea,
the treatments that were studied (Moher, Dulberg, generating hypotheses, designing investigations, col-
& Wells, 1994). In the majority of these studies, sta- lecting and analyzing data, and so on. Yet, in clini-
tistical power was weak. Of course, I cannot speak cal psychology research is conducted in laboratory
for you, but I personally am not pleased to learn that and clinical settings and addresses theoretical and
viable treatments might be available but one cannot applied issues. Other areas of psychology such as
tell because the studies were not well designed. That counseling, educational, and school psychology and
is, the sample size was too small to demonstrate a other disciplines such as psychiatry, nursing, and
real effect if there was one. Let us try to ignore for social work also engage in research that spans quite
the moment the tax dollars likely to be wasted (from diverse settings, participants, and goals. Research
federally and state-funded research), the enormous in these areas often presents novel challenges to
inconvenience and perhaps pain (physical and the investigator. Consider the diversity of topics,
psychological) of many of the participants run in samples, and settings in which clinical psychological
such trials, and the ethical issues all of this raises research is conducted.
by exposing participants to any condition when The scope of research in clinical psychology
the research might not be able to obtain an answer. is enormous. Among the topics addressed are the
We want all investigators who design studies, who assessment, diagnosis, course, treatment, and preven-
review proposals for research, or who are involved tion of social, emotional, and behavioral problems;
in the research process in some way to understand affective and cognitive processes, personality, family
methodology to minimize delays in accumulating processes, peer relationships, the interface of mental
knowledge that can affect people in everyday life. and physical health, and cross-cultural differences.
In many cases, the life-and-death features come The populations studied include children, adoles-
from the topics under study. In psychology, we want cents, adults, elderly people, and people with special
to discover the etiologies and means of prevention of experiences (e.g., homelessness, divorce, prisoners of
suicide, psychiatric disorders, addictions, dementia, war), medical impairment and disease (e.g., cancer,
and aging, to mention huge areas of research. Each AIDS, spinal cord injury, diabetes), or psychological

16
Methodology: What It Is and Why It Is so Important
Copyright American Psychological Association

disorder or dysfunction (e.g., depression, anxiety, answer questions that are theoretically important
posttraumatic stress disorder, autism). People in or lead to understanding clinical phenomena and
contact with special populations, that is, those who populations.
are exposed to someone with a special condition, are Often, clinical samples are studied and evalu-
themselves often studied (e.g., children of parents ated over a period of time, even if only a month
who abuse alcohol, spouses of depressed patients, or two. The challenges of recruiting participants;
siblings of children with physical disabilities). Yet, retaining them if the study lasts weeks, months,
the populations also include nonhuman animal sub- or years; ensuring that their care is fine if they are
jects in an effort to test basic theory, processes, or in a clinical sample; and obtaining enough partici-
experiences in the lab and to evaluate animal models pants to test the hypotheses are a few of the salient
of some state or clinical dysfunction. Research in challenges. These challenges can have implica-
clinical psychology is conducted in diverse settings tions for the conclusions and whether the conclu-
(e.g., laboratory, clinics, hospitals, prisons, schools, sions can be generalized to other investigators,
industry) and in the absence of structured settings samples, and settings. The challenges also mean
(e.g., runaway children, homeless families). Finally, that quite different methodological approaches are
research in clinical psychology is also conducted in often used, including diverse designs (e.g., group
conjunction with many other areas of research and and single case) and methods of data evaluation
different disciplines (e.g., criminology, health psychol- (e.g., statistical and nonstatistical data evaluation;
ogy, neurology, pediatrics, psychiatry, public health). Kazdin, 2016).
Understandably, diverse methods of study are My purpose in highlighting the diversity of
required to meet the varied conditions in which clinical psychology is to underscore the importance
clinical psychologists work and the special chal- of facility with the methods of research. Special
lenges in drawing valid scientific inferences from demands or constraints are frequently placed on the
these situations. The methodological diversity of clinical researcher. Ideal methodological practices
clinical research, as with the substantive diversity, (e.g., random assignment) are not always available,
can be illustrated in many different ways. Studies but they are not always necessary. Also, restrictions
vary in the extent to which the investigator can may limit the researcher’s options (e.g., a control
exert control over the assignment of participants to group might not be feasible, only small sample sizes
conditions or administration of the intervention and may be available). If one is conducting research
the selection of preexisting groups (with a type of that requires more than one occasion (e.g., repeated
condition or experience) and how they are followed assessments on two or more occasions, intervention
and evaluated. studies, longitudinal studies on the course of clinical
Occasionally, clinical psychologists conduct dysfunction), retaining participants in the project
research with college students recruited from and loss of participants (attrition) present new chal-
introductory psychology classes at a university. lenges (e.g., for data analyses).
Participants are seen for a session or two and com- The task of the scientist is to draw valid infer-
plete a laboratory task. If, however, one looks at the ences from the situation and to use methodology,
premier journals in clinical psychology and allied design, and statistics toward that end. In clini-
disciplines (e.g., Journal of Abnormal Psychology, cal psychology and related areas of research, the
Journal of Consulting and Clinical Psychology, options in methodology, design, and statistics
Clinical Psychological Science), it becomes clear must be greater than in more basic research areas
that this is not the usual paradigm for research. to permit the investigator to select and identify
And yet, many basic questions about phenomena of creative solutions. Clinical research is not in any
interest (e.g., emotion regulation, psychopathy, and way soft science; indeed, the processes involved in
brain activation in relation to varied social cues that clinical research reflect science at its best precisely
require carefully controlled conditions in the lab) because of the thinking and methodological inge-
might well be studied with nonclinic samples and nuity required to force nature to reveal its secrets.

17
Alan E. Kazdin Copyright American Psychological Association

Deploying strategies to accomplish this requires an Part III, Sampling and Assigning Participants
appreciation of the purposes of research and the to Conditions, includes chapters that address pro-
underpinnings of research strategies in addition to cedures, practices, and decisions as one moves
the concrete practices on which methodology draws. from research ideas to concrete details of the study.
Sampling and whom to include as participants can
make a huge difference, and current issues (related
GOALS AND FOCUS OF THIS BOOK
to the use of college students as participants and
The goal of this book is to help the reader design, online participant pools) are covered. Also, the
conduct, recognize, and appreciate high-quality importance of and limitations of random assignment,
research. High-quality research begins with the a core feature of experiments, are addressed. Finally,
idea, theory, or prediction that underlies the study different design options are detailed, from experi-
and its contribution to knowledge. The quality of ments in which conditions are manipulated to obser-
the research continues to be defined by how that vational studies in which groups are selected because
idea is translated into an investigation. Addressing of some particularly important characteristic.
potential problems and solutions, as highlighted Part IV, Assessment, presents scale evaluation
earlier, contributes to how well that idea has been and development. Although most research work
translated. Can the investigator draw the conclu- does not involve development of scales, selecting
sions he or she wants on the basis of how the study measures for research is a critical step. Articles on
was planned, executed, and evaluated? High-quality scale development and measurement reliability and
research occurs when scientists are skilled in each validity convey key issues that influence or ought to
of these phases of research. This book addresses the influence the measures one selects and the interpre-
full process of research, moving from ideas, design, tation of research results. Issues raised in the assess-
assessment, data evaluation, interpretation, and pre- ment of underrepresented groups are also included
sentation to communication of findings. Pervasive in in this part. Ethnic, minority, and cultural issues are
all facets of research are ethical issues and responsi- of great interest in their own right because science
bilities to the individual participants but also to the and psychological science are designed to under-
scientific enterprise more generally. The chapters in stand everyone and how differences and similarities
this book address multiple facets and components of emerge. In addition, the study of diverse groups
research and the flow from planning, to executing, raises important methodological issues related to
and to evaluating a study. sampling, measurement validation, and generality of
Part I of this book, Introduction: Overview and results.
Background, conveys what methodology is and the Part V, Data Analysis, Evaluation, and
roles it plays in scientific knowledge. Perhaps the Presentation, includes several articles that relate
most critical point is to conceive of methodology not data evaluation to other facets of research. Major
only as a set of practices but as a way of approaching attention is given to statistical significance testing,
the subject matter of interest. The goal of research is including its origins, strengths, and limitations.
to draw valid inferences, and methodology provides There is deep concern regarding the uses and mis-
the means to accomplish this. uses of statistical significance and null-hypothesis
Part II, Beginning the Research Process, focuses testing. The chapters in this part encompass critical
on sources of ideas for research and different areas concepts and statistics, including statistical power,
in which the process of developing a study might effect size, and alternatives to null-hypothesis test-
begin. Key concepts that often guide research (e.g., ing. Data presentation is also covered in a chapter
correlates, risk factors, mediators) and various ways on novel and standard ways of graphing one’s data.
to study a phenomenon (e.g., basic laboratory to Part VI, Special Topics: Evaluation in Clinical
applied research) are discussed. The material is Practice and Research, presents assessment in ways
designed to help the researcher identify the problem that unify clinical practice and research. The impor-
or topic of interest and how it might be studied. tance, contribution, and examples of evaluation in

18
Methodology: What It Is and Why It Is so Important
Copyright American Psychological Association

clinical work and qualitative measures to enhance research by increasing replications, that is, verifica-
the understanding of therapeutic change from the tion of findings by seeing whether they can be repro-
clients’ perspective are discussed. Also, assessment duced in additional studies. The renewed interest
in treatment outcome research extends the discus- stems from many influences, including concerns
sion with similar concerns, namely how to evaluate over the bias of journal publication, by which statis-
therapeutic change in ways that are meaningful to tically significant findings are favored, questionable
the client. Ethnic and cultural variation can greatly research practices that may further bias and foster
influence evaluation and interpretation of results. more chance findings in the research literature,
Methodological challenges in designing and evalu- and fraud, among other influences. Replication is
ating interventions with ethnically and culturally not merely just doing the study over again. Several
diverse groups are covered in this part. relevant methodological issues are discussed, along
Part VII, Multiple Methodologies, presents vari- with broader issues about how to foster replication
ous methodological approaches to research. The and what changes might be needed in incentives for
dominant methodological approach in research researchers—in journal publication and the review
is null-hypothesis statistical testing. Yet, other of manuscripts, for example—to increase the likeli-
approaches are equally as rigorous and scientific but hood of replication attempts.
receive much less attention. Three methodological Part X, Publication and Communication of
approaches are highlighted in this part, including Research, addresses the preparation of manuscripts
math and statistical modeling (a very user-friendly designed to communicate research. Communication
chapter), qualitative research, and single-case experi- is a logical conclusion to completion of research.
mental designs. As the book notes in a few places, There are reporting standards and guidelines for
how one studies a phenomenon can greatly influence preparing journal articles that involve empirical
as well as restrict what one learns. Diverse methodol- studies or literature reviews, and these are covered
ogies can elaborate that same phenomenon in novel in the chapters. Concrete guidelines are provided
ways not available through another approach. on preparing articles for publication that under-
Part VIII, Ethical Issues and Scientific Integrity, score how to convey the rationale for the study as
focuses on investigators’ responsibilities to par- well as many decisions along the way as part of
ticipants, science, and the public at large. This the design, assessment, and data evaluation pro-
part begins with the ethical principles and code of cedures. The chapters in this part are designed to
conduct developed by the American Psychological convey the thought processes prompted by meth-
Association as well as a brief guide from the odology that deserve attention in preparing reports
National Institutes of Health. Research ethics for of one’s own research, reviewing the literature in
ethnic and minority samples can raise special issues an area, conducting meta-analyses, and preparing
that are addressed in their own chapter. Scientific grant applications.
integrity issues are raised and include questionable Finally, in Part XI, Perspectives on Methodology,
research practices and decision making that can bias I provide closing comments to convey some of the
how research findings are presented, fraud and fab- broader lessons methodology teaches. Among the
rication of data, and allocation of credit of author- recommendations is to encourage the use of novel
ship. Ethics and scientific integrity reflect a range and diverse methods of investigation. The purpose is
of responsibilities that go beyond the sets of issues not diversity and novelty for their own sake. Rather,
illustrated in the chapters in this section, which is methods of studying a phenomenon often influence
why the overarching ethical principles and code of what can be learned and what specifically is learned.
conduct are so important to have as a reference and Complementary methods of study can elaborate the
broader guide. phenomenon in new ways. Researchers are encour-
Part IX, Reproducibility of Findings: Replication aged to study phenomena in diverse ways and to
and Improving Research Practices, focuses on develop collaborations in their careers that facilitate
renewed interest in improving our scientific this. Novel findings can come from novel methods.

19
Alan E. Kazdin Copyright American Psychological Association

Such findings also come from novel ideas, but even (2014). The extinction of the dinosaurs. Biological
novel ideas studied with the usual participants, usual Reviews, 90, 628–642. Retrieved from http://
onlinelibrary.wiley.com/doi/10.1111/brv.12128/full
measures, and usual data analyses can be constrained.
The chapters in each section raise points cen- Cohen, J. (1962). The statistical power of abnormal-
social psychological research: A review. Journal
tral to what might be considered phases of a study of Abnormal and Social Psychology, 65, 145–153.
or some temporal sequence, beginning with the http://dx.doi.org/10.1037/h0045186
research idea, of course, and then moving to details Cook, T. D., & Campbell, D. T. (1979). Quasi-
about how the study will test the idea in concrete experimentation: Design and analysis issues for field
ways, who the participants will be, what measures settings. Chicago, IL: Rand McNally.
will be used, and so on. Of course, there is an obvi- Danziger, S., Levav, J., & Avnaim-Pesso, L. (2011).
Extraneous factors in judicial decisions. Proceedings
ous sequence of steps. At the same time, many
of the National Academy of Sciences of the United
chapters connect the research process by spanning States of America, 108, 6889–6892. http://dx.doi.
different phases of research and showing their inter- org/10.1073/pnas.1018033108
connections. For example, ethical issues and data Dunning, D., & Balcetis, E. (2013). Wishful seeing
evaluation and analyses emerge at the proposal stage how preferences shape visual perception. Current
before the first participant is even run. So not all Directions in Psychological Science, 22, 33–37.
http://dx.doi.org/10.1177/0963721412463693
aspects of methodology follow a stepwise progres-
Eberhardt, J. L., Davies, P. G., Purdie-Vaughns, V. J.,
sion. Even the study itself might be conceptualized & Johnson, S. L. (2006). Looking deathworthy:
as not having a fixed starting and ending point. For Perceived stereotypicality of Black defendants pre-
example, the write-up of a study is not the end of a dicts capital-sentencing outcomes. Psychological
sequence of tasks in research. The well-described Science, 17, 383–386. http://dx.doi.org/10.1111/
j.1467-9280.2006.01716.x
and well-presented write-up ought to point rather
Haselton, M. G., & Gildersleeve, K. (2011). Can
clearly to the next studies and hence constitutes a
men detect ovulation? Current Directions in
new beginning. I have divided the book into parts, Psychological Science, 20, 87–92. http://dx.doi.
but the research process is continuous, and consid- org/10.1177/0963721411402668
erations (assessment, design, data analyses) emerge Johnson, M. K. (2006). Memory and reality. American
at multiple places (before the study is begun, when Psychologist, 61, 760–771. http://dx.doi.org/10.
the write-up is underway). 1037/0003-066X.61.8.760
We begin now with the often most daunting Kahneman, D. (2011). Thinking, fast and slow. New
York, NY: Farrar, Straus & Giroux.
research challenge, namely, developing the idea that
is to serve as the basis for a study. There are many Kazdin, A. E. (2016). Research design in clinical psy-
chology (5th ed.). Needham Heights, MA: Allyn &
guidelines and aids, and these are provided in Part II. Bacon.
Kessler, R. C., & Wang, P. S. (2008). The descriptive epi-
References demiology of commonly occurring mental disorders
Bakker, M., van Dijk, A., & Wicherts, J. M. (2012). in the United States. Annual Review of Public Health,
The rules of the game called psychological science. 29, 115–129. http://dx.doi.org/10.1146/annurev.
Perspectives on Psychological Science, 7, 543–554. publhealth.29.020907.090847
http://dx.doi.org/10.1177/1745691612459060 Krug, E. G., Dahlberg, L. L., Mercy, J. A., Zwi, A. B.,
Baumann, J., & DeSteno, D. (2010). Emotion guided & Lozano, R. (2002). World report on violence
threat detection: Expecting guns where there are and health. Geneva, Switzerland: World Health
none. Journal of Personality and Social Psychology, Organization.
99, 595–610. http://dx.doi.org/10.1037/a0020665 Maartens, R. (2013). The accelerating universe. Quest, 9,
Brainerd, C. J., & Reyna, V. F. (2005). The science 14–16.
of false memory. New York, NY: Oxford University Moher, D., Dulberg, C. S., & Wells, G. A. (1994).
Press. http://dx.doi.org/10.1093/acprof:oso/
Statistical power, sample size, and their report-
9780195154054.001.0001
ing in randomized controlled trials. JAMA,
Brusatte, S. L., Butler, R. J., Barrett, P. M., Carrano, M. T., 272, 122–124. http://dx.doi.org/10.1001/
Evans, D. C., Lloyd, G. T., . . . Williamson, T. E. jama.1994.03520020048013

20
Methodology: What It Is and Why It Is so Important
Copyright American Psychological Association

Pohl, R. F. (Ed.). (2012). Cognitive illusions: A handbook Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002).
on fallacies and biases in thinking, judgment and Experimental and quasi-experimental designs for
memory. New York, NY: Psychology Press. generalized causal inference. Boston, MA: Houghton
Mifflin.
Roediger, H. L., III, & McDermott, K. B. (2000). Tricks
of memory. Current Directions in Psychological SIDS Network. (2015). Sudden infant death syndrome
Science, 9, 123–127. http://dx.doi.org/10.1111/1467- and other infant death information web site.
8721.00075 Retrieved from http://sids-network.org
Sedlmeier, P., & Gigerenzer, G. (1989). Do studies of sta- Wells, G. L., & Loftus, E. F. (2013). Eyewitness testi-
tistical power have an effect on the power of studies? mony memory for people and events. In I. B. Weiner
Psychological Bulletin, 105, 309–316. http://dx.doi. (Ed.), Handbook of research: Forensic psychology
org/10.1037/0033-2909.105.2.309 (Vol. 11, pp. 617–629). New York, NY: Wiley.

21

You might also like