Newman 2016
Newman 2016
Newman 2016
ScienceDirect
Behavior Therapy xx (2016) xxx – xxx
www.elsevier.com/locate/bt
Please cite this article as: Michelle G. Newman, et al., Interpersonal Problems Predict Differential Response to Cognitive Versus Behavioral
Treatment in a Randomized Controlled Trial, Behavior Therapy (2016), http://dx.doi.org/10.1016/j.beth.2016.05.005
2 newman et al.
Interpersonal problems are likely candidates as relevant to treatment response (Hilbert et al., 2007;
moderators of therapy given that individuals with McEvoy, Burgess, & Nathan, 2014; Renner et al.,
the same diagnosis are often heterogeneous in 2012). Second, overall interpersonal problems pre-
terms of their predominant interpersonal difficulties dicted differential response to interventions. Higher
(e.g., Kachin, Newman, & Pincus, 2001; Przeworski overall problems predicted less improvement in
et al., 2011). A relevant, well-developed framework for depression or anxiety from group CBT, but not
measuring such problems is the interpersonal circum- from individual CBT (McEvoy et al., 2014), and
plex (IPC), which assesses a wide variety of interper- predicted greater attendance in supportive, but not
sonal characteristics and behaviors (Gurtman, 2009). interpretive, group therapy for personality disorders
The interpersonal problems IPC consists of “octant” (Ogrodniczuk, Piper, & Joyce, 2006). Lastly, sub-
scales representing underlying dysfunctions of types of interpersonal problems may predict stronger
affiliation/warmth (e.g., needing to take care of others) responses to specific therapies. For example, those
vs. coldness (e.g., seeking distance from others) and with avoidant personality disorder who had interper-
dominance (e.g., difficulty considering others’ point of sonal problems related to being cold-avoidant benefit-
view) vs. submission (e.g., excessively deferring to ted from graduated exposure, but not from skills
others). Interpersonal problems can be studied at the training (Alden & Capreol, 1993). Also, those with
level of overall problems or more specific types of more dominant problems (i.e. being too controlling)
problems via octant scales (e.g., being “socially responded more to a nonmanualized community
avoidant” is defined as being both cold and submissive, psychodynamic therapy for personality disorders,
whereas “intrusiveness” is a warm and dominant but not to manualized supportive-expressive dynamic
problem; see Figure 1). Such problems are relatively therapy (Vinnars et al., 2007). Such findings suggest
stable over time, suggesting they are trait-like charac- the possibility that the effects of interpersonal
teristics (Horowitz, Rosenberg, Baer, Ureno, & Vila- problems may depend on specific features of the
senor, 1988; Vittengl, Clark, & Jarrett, 2003). psychotherapy. However, effects found in non-CBT
Knowledge about such problems might facilitate interventions may not generalize to CBT, and there is
individualized treatment planning, given that inter- no prior research that might be used to predict how
personal problems have predicted treatment response interpersonal problems may shape differential re-
to both CBT and other therapies (e.g., psychodynam- sponse to cognitive versus behavioral therapies.
ic therapy). For example, clients’ overall pretreatment Despite a lack of direct data on interpersonal
interpersonal problems predicted less improvement moderators of cognitive versus behavioral thera-
or greater rates of dropout across individual or group pies, hypotheses may be informed by theory and
CBT for depression or anxiety as well as individual research on internalizing/externalizing coping styles
CBT or interpersonal therapy for binge eating (Beutler & Mitchell, 1981; Welsh, 1952). Those
disorder, suggesting that interpersonal problems are who “internalize” are relatively passive and with-
drawn and tend to be more interested in thinking,
whereas those with an “externalizing” style are
characteristically more active and assertive and
more interested in doing. Internalizers had greater
symptom reduction from interventions emphasizing
intellectual insight, whereas externalizers fared
better with more concrete, experiential, and
action-oriented therapies (Beutler, 1979; Beutler
& Mitchell, 1981; Beutler, Mohr, Grawe, Engle, &
MacDonald, 1991; Calvert, Beutler, & Crago,
1988; Cooney, Kadden, Litt, & Getter, 1991). For
example, alcoholic patients with higher levels of
externalizing coping features did better in response
to behaviorally focused skills training compared to
a more insight-oriented treatment (Cooney et al.,
1991; Kadden, Cooney, Getter, & Litt, 1989). Such
findings are relevant to interpersonal problems
FIGURE 1 The interpersonal circumplex with eight octants because passivity maps onto cold-submissive and
representing combinations of the dimensions of dominance and submissive octants whereas tendencies to be active
affiliation. Interpersonal problem types, which reflect rigid or and assertive map onto the dominant and friendly-
extreme versions of normal social behavior, are superposed on dominant octants of the circumplex (i.e. dominant,
these octants. and intrusive interpersonal problems; Gurtman,
Please cite this article as: Michelle G. Newman, et al., Interpersonal Problems Predict Differential Response to Cognitive Versus Behavioral
Treatment in a Randomized Controlled Trial, Behavior Therapy (2016), http://dx.doi.org/10.1016/j.beth.2016.05.005
interpersonal problems predict differential response 3
1991; Wiggins & Broughton, 1991). Furthermore, Pretreatment interpersonal problems may be
interpersonal traits and problems, although not particularly important for understanding the therapy
synonymous, reflect overlapping constructs that response of those with GAD, and ultimately tailoring
occupy the same IPC domains (Alden, Wiggins, & treatment to the individual, given evidence of robust
Pincus, 1990). Both cognitive and behavioral links between GAD symptoms and heterogeneous
interventions have shown efficacy for GAD in interpersonal difficulties (e.g., Newman & Erickson,
general (Borkovec & Ruscio, 2001); however, 2010). For example, individuals with GAD feel easily
particular interpersonal problems may lead to slighted compared to those with no diagnosis
differential responses to features of these therapies. (Gasperini, Battaglia, Diaferia, & Bellodi, 1990),
For example, one might expect that problems are disproportionately likely to be separated or
related to being domineering (overly dominant) divorced (Afifi, Cox, & Enns, 2006; Grant et al.,
and intrusive (excessive warm-dominance; Alden 2005), and endorse higher relational conflict than
et al., 1990) would predict a better response to couples with an agoraphobic member (Friedman,
concrete, action-oriented, and experiential treat- 1990). They report higher interpersonal disturbance
ments (e.g., “pure” relaxation-based behavioral in most octants compared to healthy controls and
intervention) over those that feature highly intellec- those with other anxiety disorders (Gamez, Watson,
tual or cognitively focused interventions (i.e. cogni- & Doebbeling, 2007), but are diverse in their
tive restructuring). This notion is consistent with predominant problems (Przeworski et al., 2011;
the finding that being higher on intrusive problems Salzer et al., 2008).
predicted greater symptom improvement from a Only two studies have examined whether inter-
behavioral weight loss program that included a personal problem subtypes predicted the impact of
low-calorie diet, skills training, and a fitness regimen psychotherapy on GAD symptoms. Crits-Christoph
(Lahmann et al., 2011). and colleagues (2004) found that higher overly
Additionally, these ideas dovetail with research on nurturant (excessively warm) problems predicted
“reactance,” a traitlike style of coping with others’ poorer response to supportive-expressive dynamic
social influence. Individuals high in “reactance” are therapy for GAD. Another study examined client
characterized by the motivation to maintain interpersonal problems in the context of cognitive
self-determination and sensitivity toward perceived and behavioral therapies for GAD (Borkovec,
threatened autonomy (Beutler, 1979). They tend to be Newman, Pincus, & Lytle, 2002). Collapsing data
more disposed to resist external influence and show across cognitive therapy (CT), behavioral therapy
less symptom improvement from interventions that (BT; applied relaxation plus self-control coping
could be perceived as containing direct challenge such desensitization [SCD]—a variant of systematic de-
as cognitive therapy, but fare better from interven- sensitization that includes positive coping imagery),
tions that allow for more self-guided coping (Beutler and combined treatment (CBT), this study reported
et al., 1991; Beutler, Harwood, Michelson, Song, & zero-order correlations between pretherapy interper-
Holman, 2011; Beutler, Machado, Engle, & Mohr, sonal problem subscale scores and a categorical
1993). For example, patients with depression who endstate functioning outcome measure. Those with
were lower in reactance responded better to cognitive more problems related to being domineering, intru-
therapy compared to those higher in reactance who sive, or vindictive at pretreatment had lower endstate
responded best to supportive self-directed therapy measures at 6-month follow-up, but there were no
(Beutler et al., 1993). Dominance has been concep- associations between pretreatment scores and end-
tualized as highly related to reactance (Beutler et al., state measures at posttreatment, 1-year, or 2-year
1991), and is a key correlate of reactance (Dowd, follow-up. Several interpersonal problems remaining
Wallbrown, Sanders, & Yesenosky, 1994); high at posttreatment also predicted endstate measures at
dominance (domineering) interpersonal problems posttreatment and 6-month follow-up (vindictive,
(e.g., the item, “It’s hard for me to take instructions intrusive, domineering, exploitable, nonassertive,
from people who have authority over me”) represent and overly nurturant). However, this study did not
a similarly strong need for autonomy (Wiggins & formally test whether interpersonal problems mod-
Broughton, 1991). In terms of CBT, it is possible that erated the impact of treatment type on outcomes.
those with dominant and intrusive interpersonal Also, the analytic strategy used by Borkovec and
problems may receive the greatest symptom improve- colleagues (zero-order correlations) failed to take
ment from treatments that offer concrete, behavioral, into account the nested nature of the data (repeated
and experiential skills that these individuals may measures nested within participants), and the use of a
perceive as relatively less challenging of their need categorical dependent measure likely limited power.
for control than cognitive therapy and can quickly Moreover, like other studies testing interpersonal
“own” themselves. problems as predictors of treatment response, these
Please cite this article as: Michelle G. Newman, et al., Interpersonal Problems Predict Differential Response to Cognitive Versus Behavioral
Treatment in a Randomized Controlled Trial, Behavior Therapy (2016), http://dx.doi.org/10.1016/j.beth.2016.05.005
4 newman et al.
GAD studies made no theory-based predictions for may have greater difficulty with the CT focus on
what types of clients might respond best to particular cognitive/intellectual processes compared to those
treatments. lower in dominance/intrusiveness. In contrast, the
The goal of the current study was to examine relatively more experiential, and self-directed,
interpersonal problems as moderators of therapy relaxation-based approach of BT focuses more on
outcome using multilevel models and to predict a concrete behaviors (i.e. relaxation and self-control
dimensional GAD symptom outcome measure. To desensitization) that clients can more quickly own
date, no GAD studies have examined how particular and execute independently and does not contain any
interpersonal problems might moderate the effect of cognitive challenge. Thus, this approach may be
cognitive and behavioral therapies on symptom experienced as relatively less autonomy-challenging
improvement. Prior findings suggest that individuals by high dominant/intrusive individuals. On this
higher in dominance and intrusiveness might experi- theoretical basis, we expected high dominance/
ence greater symptom reduction in a purely behavioral intrusiveness individuals to respond better to BT
relaxation-based treatment compared to cognitive (SCD) vs. treatments incorporating CT.
therapy or a combination of these treatments. The In summary, we made targeted predictions for
present study tests this theory-derived prediction in the the domineering (high dominance) and intrusive
context of a secondary analysis of data from Borkovec (high dominance plus affiliation) octants given that
et al. (2002), which compared a behavioral treatment personality characteristic associated with an exter-
that combined applied relaxation with self-control nalizing coping style and reactance have mapped
coping desensitization (BT), cognitive therapy (CT), onto these regions of the circumplex; analyses for
and a combined treatment (CBT) and found no dif- other octants were thus considered exploratory.
ferences in their efficacy for GAD at any time point.
The protocol for CT in this study explicitly empha- Method
sized intellectual analysis and Socratic challenge of participants
clients’ perspectives. In contrast, the protocol for BT Four hundred and fifty-nine people responded to
emphasized experiential participation in relaxation local newspaper advertisements or referrals from
exercises (Bernstein & Borkovec, 1973), clients’ mental health practitioners. Of these, 320 were
self-guided development of coping skills related to ruled out by phone screens for not meeting study
“letting go” via relaxation, and clients’ self-directed inclusion criteria, 54 clients were ruled out via an
desensitization to feared images with imagined initial structured interview, and 9 clients were ruled
positive coping at their own pace (Goldfried, 1971). out during a second structured interview, leaving
We predicted an interaction such that individuals 76 participants with primary generalized anxiety
with problems related to being excessively dominant disorder who entered treatment. However, 7 clients
(domineering) and affiliative-dominant (intrusive) dropped out at early stages of treatment (4 in BT, 2
would respond better to relaxation-based BT than in CT, and 1 in CBT), leaving 69 clients who
treatments incorporating CT (either CBT or CT completed treatment. Because the IIP–C was added
alone). This prediction was based on findings that to the assessment battery near the end of the 2 nd
domineering and intrusive octants are characterized year of the project, 47 people provided data at
by assertiveness and a need for autonomy (Wiggins baseline for the current study (CT n = 15; BT n = 16;
& Broughton, 1991). Traits related to assertiveness CBT n = 16). Participants ranged in age from 18 to
have predicted better response to more concrete, 65 years (M = 39.11, SD = 12.31). Of the sample,
action-oriented treatments than to treatments pro- 63.8% were female. The sample was 89.4%
moting intellectual understanding (Beutler et al., Caucasian, 4.3% Latino, 4.3% Indian, and 2.1%
1991). Similarly, people with a strong need to feel in African American. Approximately 10% were tak-
control have responded more poorly to cognitive ing psychotropic medications.
therapy for depression compared to other therapies
(Beutler et al., 2011). On one hand, cognitive and procedure
behavioral therapists have both been rated as Selection and Assessor Outcome Ratings
directive, but also as supportive and empathic— Admission criteria included consensus between the
contrary to a stereotype of CBT as cold (Keijsers, two diagnostic interviewers on: a principal diagno-
Schaap, & Hoogduin, 2000). However, although sis of GAD, no diagnosable panic disorder (as
CT and BT may be equivalently “directive,” high recommended by the funding agency's review
dominance/intrusive individuals may be more sensi- committee), a Clinician’s Severity Rating (CSR)
tive to misinterpreting cognitive restructuring as a for GAD of 4 (moderate) or more, absence of
direct challenge given the “devil’s advocate” posi- concurrent psychosocial therapy, no history of
tion that therapists often take and such clients having received CBT methods in prior therapy, no
Please cite this article as: Michelle G. Newman, et al., Interpersonal Problems Predict Differential Response to Cognitive Versus Behavioral
Treatment in a Randomized Controlled Trial, Behavior Therapy (2016), http://dx.doi.org/10.1016/j.beth.2016.05.005
interpersonal problems predict differential response 5
medical contributions to the anxiety, no antidepres- 16-item self-report measure of pathological worry.
sant medication, stable dosage of any psychotropic It has high internal consistency (Meyer et al., 1990;
medications, and absence of severe major depressive .83 in the current sample), retest reliability ranging
disorder, substance abuse, psychosis, and organic from .74–.93, as well as strong convergent and
brain syndrome. All but two clients (97.1%) met discriminant validity (Molina & Borkovec, 1994).
both DSM-III-R and DSM-IV criteria for GAD. The Inventory of Interpersonal Problems–
Advanced clinical graduate students trained to Circumplex Scale (IIP-C; Alden et al., 1990) is a
reliability in diagnostic interviewing administered 64-item measure of interpersonal problems typically
30-minute phone screens and the Anxiety Disorders reported by clients seeking psychotherapy. Items
Interview Schedule–III–R (ADIS-R; Di Nardo & reflect both behavioral deficiencies and behavioral
Barlow, 1988). Those not ruled out received the excesses. Thirty-nine items are phrased as “It is hard
ADIS-R, which included the Hamilton Anxiety for me to . . . ” followed by, for example, “say 'no' to
Rating Scale (HARS; Hamilton, 1959), CSR for other people.” The remaining 25 items are phrased as
GAD, and additional questions corresponding to two “These are things I do too much: . . . ” followed by, for
GAD criteria being proposed at the time of study example, “I open up to people too much.” Items are
initiation by the Diagnostic and Statistical Manual of rated on a 5-point Likert scale from 0 (not distressed
Mental Disorders (4th ed.; DSM–IV; American at all about this problem) to 4 (extremely distressed
Psychiatric Association, 1994) subcommittee for about this problem). Eight 8-item subscales corre-
GAD (i.e. uncontrollable worrying, and three of six sponding to octants on the circumplex make up the
associated symptoms). A second ADIS-R was admin- IIP-C, including domineering (e.g., “I try to control
istered within 2 weeks by the therapist who would see other people too much”), and intrusive (e.g., “It is
the client in therapy to reduce likelihood of false hard for me to stay out of other people’s business”)
positive cases. Pretreatment diagnoses were based on problems, which were the primary variables of
consensus between the independent structured inter- interest in the current study. Other subscales include
viewers. A random subsample of 20% of pretreat- vindictive (e.g., “I try to get revenge on other people
ment audiotapes of ADIS-R interviews conducted by too much”), cold (e.g., “It is hard for me to feel close
the primary assessor (prior to developing consensus) to other people”), socially avoidant (e.g., “It is hard
was reviewed for reliability purposes. For the for me to socialize with other people”), nonassertive
presence of GAD, kappa agreement was 1. Outcome (e.g., “It is hard for me to let other people know what I
measures were administered at pre, posttreatment, want”), exploitable (e.g., “It is hard for me to feel
6-month, 12-month, and 24-month follow-ups. angry at other people”), and overly nurturant (e.g., “I
try to please other people too much”). Retest
measures reliability (total r = .98; average subscale r = .81)
The CSR (Di Nardo & Barlow, 1988) is a 0 (none) to and internal consistency for the octant scales (α =
8 (very severely disturbing/disabling) score assigned .72–.85) have been demonstrated (Horowitz et al.,
by interviewers to reflect degree of impairment 1988), and ranged from α = .73–.89 in this study.
associated with each disorder. Interrater reliability Client improvement as measured by the IIP correlated
of CSRs in the current study ranged from an with improvement on symptom measures, and with
intraclass correlation (ICC) of .77 to 1. assessments of independent observers (Horowitz
The State Trait Anxiety Inventory–Trait Version et al., 1988). Further, the scale predicted the types
(STAI-T) is 20-item scale measure of trait anxiety of interpersonal issues discussed in therapy (Horowitz
with high internal consistency reliability (.86 in the et al., 1988; Renner et al., 2012).
current sample), good retest reliability (high .70's),
and strong convergent and discriminant validity therapy conditions
(Spielberger, Gorsuch, Lushene, Vagg, & Jacobs, Participants were randomly assigned to receive
1983). either BT (N = 16), CT (N = 15), or combined CBT
The HARS (Hamilton, 1959) is a 14-item clinician- (N = 16). In all conditions, therapy manuals were
administered scale of severity of anxious symptoms. used. The first four sessions were 2 hours in
Internal consistency ranged from adequate to good duration; remaining sessions were 1.5 hours. The
(α = .77 to .81 [Moras, di Nardo, & Barlow, 1992]; first 30 minutes of each BT and CT session involved
.82 in the current sample). Interrater reliability only supportive listening (SL) to equalize therapist
ranged from an ICC of .74 –.96 (Bruss, Gruenberg, contact time (see Borkovec et al., 2002, for more
Goldstein, & Barber, 1994; ICC = .89 in the present detailed description of the therapy).
study). Fourteen weekly sessions were administered.
The Penn State Worry Questionnaire (PSWQ; Several aspects were common to the three conditions:
Meyer, Miller, Metzger, & Borkovec, 1990) is a presentation of a model of anxiety and rationale for
Please cite this article as: Michelle G. Newman, et al., Interpersonal Problems Predict Differential Response to Cognitive Versus Behavioral
Treatment in a Randomized Controlled Trial, Behavior Therapy (2016), http://dx.doi.org/10.1016/j.beth.2016.05.005
6 newman et al.
therapy, self-monitoring and early identification of Similar to other treatment studies (Newman et al.,
anxiety cues, homework assignments, and review of 2011), we created a single continuous variable to
homework. CT entailed logical analysis, examina- represent GAD symptom severity. A composite
tion of evidence and probabilities, labeling logical provides a more valid measure of psychopathology
errors, decatastrophizing, generation of alternative and a means of reducing experiment-wise error rate
thoughts and beliefs, plus SL. BT entailed progres- (Horowitz, Inouye, & Siegelman, 1979). The com-
sive, cue-controlled, and differential relaxation posite included the CSR, PSWQ, the STAI-T, and the
training as described in Bernstein and Borkovec HARS. Each scale was first standardized based on
(1973), slowed diaphragmatic breathing, relaxing sample means and standard deviations, and then
imagery, meditational relaxation, applied relaxation scales were averaged to create a standardized com-
training, self-control desensitization as described by posite. Positive values of the composite (i.e. above the
Goldfried (1971), and SL. For self-control desensiti- sample mean) represent more anxiety pathology, and
zation, clients constructed anxiety cue hierarchies. negative values (i.e. below sample mean) represent less
They practiced relaxation, and when deeply relaxed, anxiety pathology. As such, negative regression
they imagined being in the presence of an external or estimates at post and follow-up reflect a beneficial
internal anxiety cue until they noted the presence of effect of treatment. The IIP-C was scored using
anxious feelings. They then continued imagining the subscales as opposed to the IPC structural summary
external situation while imagining deploying coping method (Wright, Pincus, Conroy, & Hilsenroth,
responses. At the elimination of anxious feelings, 2009) for two reasons. First, subscale scores are easier
clients imagined continued coping deployments for for practicing therapists to compute than structural
20 s and then discontinued imagery and focused only summary scores. Secondly, subscale scores permit
on the relaxed state for 20 s. Scenes were repeated examination of specific types of problems dimension-
until clients could no longer generate anxiety or ally, consistent with our theory-based predictions and
were able to eliminate it rapidly (i.e. within 5–7 s). are better suited for inferential statistical tests.
Homework emphasized frequent applications of Results were analyzed using multilevel models.
relaxation and focus on living in the present moment. Time was modeled in a piecewise manner, such that
CBT contained all of the treatment techniques in CT pre-post time and post-follow-up were both included
and BT, but had no separate SL segment included. in the same models, but were modeled with different
terms. Piecewise models allowed for the symptom
Planned Analyses change to be different from pre- to posttreatment
There was no missing outcome data and therefore, than from posttreatment to follow-up, given that one
no data replacement strategies were used. Power would expect different rates of change during the two
analyses were calculated for each of the a priori time periods. Post-follow-up time used one regres-
models based on Monte-Carlo simulation studies of sion coefficient containing three follow-up times
the fixed and random effect model estimates. (6-month, 1-year, and 2-year) as a single continuous
Simulation studies represent a gold standard in predictor. Each moderator was considered separate-
power analyses calculations (Ma, Thabane, Beyene, ly to ensure that the models had adequate degrees of
& Raina, 2016; Muthén & Muthén, 2002). After freedom and that multicollinearity was not an issue
conducting 1,000 simulations per condition, power (Leal, Bean, Thomas, & Chaix, 2012). However,
estimates for the three-way interaction between given our a priori hypotheses, domineering and
time, the moderator variable, and therapy condi- intrusiveness as moderators were modeled first and
tion as well as simple slopes analyses suggested that were viewed as the primary analyses, whereas
each test had requisite power. 1 examination of additional IIP-C subscales as moder-
ators were considered exploratory. Each multilevel
1
Power = 95.55% for the interaction between pre-post time, model was analyzed in the R package lme4 (Bates,
intrusiveness, and condition; power = 90.29% for the interaction with Maechler, & Bolker, 2012). Random effects includ-
post-follow-up, time, intrusiveness, and condition; power = 91.72% ed intercepts, and time slopes (pre-post and
for the interaction between post-follow-up time, domineering, and post-follow-up) as nested within persons. The
condition. Power = 93.40% for differences in slopes between BT and
CBT and 79.10% for differences in slopes between CT and BT for the
covariance structure for random effects was unstruc-
interaction between pre-post time, intrusiveness, and condition; tured. Fixed effects included (1) the main effects of
power = 74.26% for differences in slopes between BT and CT and time (pre-post and post-follow-up), the moderator
85.70% for differences in slopes between BT and CBT for the variable of interest (IIP subscale), and condition;
interaction between post-follow-up time, intrusiveness, and condition; (2) two-way interactions between time (pre-post or
power = 86.03% for differences in slopes between BT and CBT for
post-follow-up time, domineering, and condition. Power = 77.73%
post-follow-up), the moderator variable of interest
for differences in slopes between BT and CT for post-follow-up time, (IIP subscale), and condition; and (3) the three-way
domineering, and condition. interaction between time, the moderator variable of
Please cite this article as: Michelle G. Newman, et al., Interpersonal Problems Predict Differential Response to Cognitive Versus Behavioral
Treatment in a Randomized Controlled Trial, Behavior Therapy (2016), http://dx.doi.org/10.1016/j.beth.2016.05.005
interpersonal problems predict differential response 7
Table 1
Correlations and Descriptive Statistics of the Inventory of Interpersonal Problems Subscales at Pretreatment
IIP Subscale 1 2 3 4 5 6 7 8
1. Vindictive 1
2. Cold .622 ⁎⁎ 1
3. Socially avoidant .235 .544 ⁎⁎ 1
4. Unassertive .273 .364 ⁎⁎ .579 ⁎⁎ 1
5. Exploitable .213 .269 .514 ⁎⁎ .829 ⁎⁎ 1
6. Overly nurturant .386 ⁎⁎ .092 -.007 .388 ⁎⁎ .538 ⁎⁎ 1
7. Intrusive .449 ⁎⁎ .142 -.051 .226 .356 ⁎ .617 ⁎⁎ 1
8. Domineering .522 ⁎⁎ .174 -.113 .022 .144 .595 ⁎⁎ .685 ⁎⁎ 1
Mean 8.94 9.62 13.90 18.56 16.98 15.29 11.80 8.40
SD 4.68 4.91 6.40 6.78 5.59 5.57 6.39 5.10
Range 0-20 3-25 2-29 4-32 6-29 1-28 1-28 0-20
Note. IIP = Inventory of Interpersonal Problems.
⁎⁎ Correlation is significant at the 0.01 level (2-tailed).
⁎ Correlation is significant at the 0.05 level (2-tailed).
interest, and condition. We first tested full models from .083–.964; ds ranged from .078–.673). Thus,
with all interactions and main effects for domineer- most interpersonal problems were distributed across
ing and intrusive problems, to examine our primary all conditions, permitting tests of differential treat-
moderation hypotheses (i.e. three-way interactions), ment response by problems. There were also no
followed by exploratory tests of other IIP subscales. differences between therapy conditions at baseline
To limit redundancy, we removed nonsignificant on the composite of GAD symptom outcome
random effects and then nonsignificant fixed effects measures F(2, 45)= 0.275, p = .640, d = .303.
from the model using the R package lmertest Examination of the distributions of the IIP-C scales
(Kuznetsova & Brockhoff, 2014). 2 To ensure high within each therapy condition suggested a normal
resolution in examining interpersonal problems, distribution. Additionally, Shapiro-Wilk tests on
subscales were used as continuous predictors in the each of the IIP-C scales for each condition were not
analyses. significant (p N .050), suggesting normal distribu-
Following each analysis, the significance of each tions. There were also no differences between those
interaction was investigated with reference to each who did or did not complete the IIP-C on GAD
group via post-hoc tests using the R package, phia symptoms at baseline, post, 6-month, 12-month, or
(Rosario-Martinez, 2013). All of the primary fixed 24-month follow-up (ps ranged from .497–.895; ds
effect coefficients’ effect sizes were converted to ranged from -0.032–0.166). Completing the IIP-C
Cohen’s d, using the following equations for (χ 2 = 0.349, p = .840) was also not significantly
F-statistics,
pffiffiffi t-statistics, and chi-squared statistics:
related to treatment condition. Additionally, com-
d¼p 2 ffiffiffiffiffiffiffiffiffi
F
, d ¼ p2ffiffiffiffiffiffiffiffiffi
t
, and d = ((4 χ 2)/(N- χ 2)) 1/2 pleting the IIP-C did not significantly moderate or
ðN −1Þ ðN −1Þ
predict outcome (ps ranged from .162–.926, ds
(Dunst, Hamby, & Trivette, 2004; Wolf, 1986). ranged from 0–0.452).
Results
descriptive statistics and baseline primary analyses
differences Moderation of Treatment Response by Intrusive
Table 1 provides descriptive statistics and correla- Problems
tions between subscales of the IIP-C at baseline. Supporting our hypotheses in the pre-post moder-
Consistent with other studies, octants that are ation analyses, there was a significant three-way
closer together on the circumplex were more highly interaction between pre-post time, condition, and
correlated with one another than more distant intrusiveness, F(2, 182) = 3.783, p = .024, d = .58.
octants. There were no significant pretreatment Significant differences emerged in the slopes be-
differences between the three compared psycho- tween BT and CBT (ΔB = 0.377, χ 2 = 7.179, p =
therapy conditions on IIP-C subscales (ps ranged .022, d = 1.098), but not in the slopes between CT
and CBT (ΔB = 0.075, χ 2 = 0.237, p = .626, d =
0.178) or BT and CT (ΔB = 0.302, χ 2 = 3.468, p =
2
Note that the full models without removing nonsignificant
.125, d = 0.710). This interaction showed that as
effects led to identical results in their significance and their direction baseline intrusiveness levels increased, GAD symp-
for the primary analyses. toms were more likely to show improvement in
Please cite this article as: Michelle G. Newman, et al., Interpersonal Problems Predict Differential Response to Cognitive Versus Behavioral
Treatment in a Randomized Controlled Trial, Behavior Therapy (2016), http://dx.doi.org/10.1016/j.beth.2016.05.005
8 newman et al.
exploratory analyses
There were no significant interactions (p N .05)
between any of the remaining interpersonal problem
subscales and either time or treatment condition,
suggesting that these variables neither predicted
change from therapy nor moderated outcome.
There was a main effect only of overly nurturant
problems (B = 0.45, SE = 0.216, t[47] = 2.071, p =
.044, d = 0.610) on anxiety at baseline, as well as at
post and all follow-up points, regardless of treatment
condition, suggesting a consistent link of these
problems and GAD symptoms.
FIGURE 2 This graph depicts the three-way interaction Discussion
between the pretreatment and posttreatment time, condition,
and intrusiveness when predicting the change in generalized As predicted, compared to participants lower on
anxiety disorder symptoms. Note that the change scores were intrusiveness, those who were relatively higher on this
calculated from the regression estimates. BT = behavioral therapy, interpersonal problem responded with greater change
CT = cognitive therapy, CBT = cognitive-behavioral therapy. in GAD symptoms to BT than to CBT at
Please cite this article as: Michelle G. Newman, et al., Interpersonal Problems Predict Differential Response to Cognitive Versus Behavioral
Treatment in a Randomized Controlled Trial, Behavior Therapy (2016), http://dx.doi.org/10.1016/j.beth.2016.05.005
interpersonal problems predict differential response 9
Please cite this article as: Michelle G. Newman, et al., Interpersonal Problems Predict Differential Response to Cognitive Versus Behavioral
Treatment in a Randomized Controlled Trial, Behavior Therapy (2016), http://dx.doi.org/10.1016/j.beth.2016.05.005
10 newman et al.
Results of our secondary, exploratory analyses Across prior treatment studies outside of GAD
revealed that no other interpersonal problems research, problems related to uninvolvement or
predicted or moderated change from psychotherapy. low affiliation typically predicted worse outcomes
However, those GAD individuals higher on overly in CT or CBT (Hardy et al., 2001; McEvoy, Burgess,
nurturant interpersonal problems at baseline exhib- & Nathan, 2013) or psychodynamic therapy
ited higher levels of GAD severity at baseline—a (Lorentzen & Høglend, 2004), whereas affiliation
relationship that was maintained across postas- and affiliative problems have generally predicted
sessment and all follow-up points. Previously, better response to CBT (Lahmann et al., 2011) and
Borkovec et al. (2002) found that overly nurtur- psychodynamic therapies (Lorentzen & Høglend,
ant problems (among other problems) remaining 2004). In contrast, the fact that for GAD studies,
after treatment predicted posttreatment and 6-month affiliative problems predicted worse treatment out-
follow-up outcomes. Similarly, in Crits-Christoph et come for psychodynamic therapy (Crits-Christoph
al. (2004) higher overly nurturant problems predict- et al., 2004) and covaried with GAD symptoms across
ed less change at postassessment in an open trial of time despite CBT suggests that extant treatments have
GAD clients who received supportive-expressive not adequately targeted pathologically affiliative
psychodynamic therapy. problems occurring in GAD. However, the foregoing
Thus, although we did not replicate a predictive theory is speculative and warrants further empirical
effect of overly nurturant problems on treatment testing to determine if it may contribute to under-
response, the significant covariation between overly standing GAD treatment response.
nurturant interpersonal problems and GAD severity Several limitations for this study should be
points to an emerging problem area in GAD mentioned. First, the study sample was highly
phenomenology, one that likely merits further educated and not very ethnically diverse; thus our
consideration, as overly nurturant problems may findings may not generalize to more diverse samples.
play a role in maintaining GAD symptoms. Although Second, because the IIP-C was added to the study
heterogeneous interpersonal problems occur in the only at the end of the second year, we had data from
context of GAD (e.g., Przeworski et al., 2011), only a subsample of the total original sample of this
several studies have linked excessive or maladaptive study and cell sizes for the three therapy conditions
forms of affiliation to GAD and/or worry. Individ- were relatively small. Although we had sufficient
uals with GAD reported childhood memories of power to test our hypotheses, it would be important
“role-reversal” or taking care of parents (Cassidy, to replicate our findings using larger and more
Lichtenstein-Phelps, Sibrava, Thomas, & Borkovec, diverse GAD sample. At the same time, our main
2009) and believe that worrying means that one findings were replicated across multiple time phases
“cares” (Hebert, Dugas, Tulloch, & Holowka, (pre-post, versus post–6 months, 1 year, 2 years),
2014); people with GAD may have learned to which is an unusual strength of this dataset as most
manage potential and actual stressors (e.g., loss of a studies examining moderation do not find the same
parent; Torgersen, 1986) by both the internal results at multiple time points. Furthermore, in
strategy of worry and interpersonal strategies related order to control for and equalize therapist contact,
to caretaking behavior (Newman & Erickson, 2010). while keeping the amount of cognitive and behav-
In line with this theory, a recent series of studies ioral therapies received equivalent across therapy
showed that after controlling for depression symp- conditions, both the BT and CT conditions contained
toms and social anxiety, worry uniquely predicted a distinct supportive listening component, whereas
perceiving one’s own interpersonal tendencies as the CBT condition did not. However, given that
affiliative on self-reported traits, interpersonal prob- our results were more similar between CT and CBT
lems, and social behavior during daily experience than between CT and BT, this likely suggests that
sampling over 1 week, and interpersonal goals in supportive listening did not contribute to the
relationship to a significant other (Erickson et al., moderation effects we found. Lastly, our measure
2016). However, worry also uniquely predicted of interpersonal problems measured dominance vs.
being viewed as unaffiliative by significant others, submissive problems, but did not directly measure
in line with previous research finding a disconnect autonomy-seeking, relevant to previous theorizing
between interpersonal self-perceptions and percep- about reactance as a treatment moderator. The
tions by others in high-worry individuals (Erickson Structural Analysis of Social Behavior (SASB) is a
& Newman, 2007). Affiliative behavior that is validated behavioral coding system and self-report
viewed by others as cold rather than genuinely measure that can examine therapist “control”
supportive could plausibly impair relationships, versus “autonomy-granting,” as well as the extent
consistent with evidence of relational difficulties in of therapist affiliation; similarly, the SASB can
GAD (Newman & Erickson, 2010). be used to code extent of client submission vs.
Please cite this article as: Michelle G. Newman, et al., Interpersonal Problems Predict Differential Response to Cognitive Versus Behavioral
Treatment in a Randomized Controlled Trial, Behavior Therapy (2016), http://dx.doi.org/10.1016/j.beth.2016.05.005
interpersonal problems predict differential response 11
Please cite this article as: Michelle G. Newman, et al., Interpersonal Problems Predict Differential Response to Cognitive Versus Behavioral
Treatment in a Randomized Controlled Trial, Behavior Therapy (2016), http://dx.doi.org/10.1016/j.beth.2016.05.005
12 newman et al.
Friedman, S. (1990). Assessing the marital environment of Kadden, R. M., Cooney, N. L., Getter, H., & Litt, M. D.
agoraphobics. Journal of Anxiety Disorders, 4, 335–340. (1989). Matching alcoholics to coping skills or interactional
http://dx.doi.org/10.1016/0887-6185(90)90030-D therapies: Posttreatment results. Journal of Consulting and
Gamez, W., Watson, D., & Doebbeling, B. N. (2007). Clinical Psychology, 57, 698–704. http://dx.doi.org/10.
Abnormal personality and the mood and anxiety disorders: 1037/0022-006X.57.6.698
Implications for structural models of anxiety and depres- Keijsers, G. P. J., Schaap, C. P. D. R., & Hoogduin, C. A. L.
sion. Journal of Anxiety Disorders, 21, 526–539. http://dx. (2000). The impact of interpersonal patient and therapist
doi.org/10.1016/j.janxdis.2006.08.003 behavior on outcome in cognitive-behavior therapy: A
Gasperini, M., Battaglia, M., Diaferia, G., & Bellodi, L. (1990). review of empirical studies. Behavior Modification, 24,
Personality features related to generalized anxiety disorder. 264–297.
Comprehensive Psychiatry, 31, 363–368. http://dx.doi.org/ Kuznetsova, A., & Brockhoff, P. B. (2014). Package ‘lmerTest’.
10.1016/0010-440X(90)90044-S Lahmann, C., Henrich, G., Henningsen, P., Baessler, A.,
Goldfried, M. R. (1971). Systematic desensitization as training Fischer, M., Loew, T., . . . Pieh, C. (2011). The impact of
in self-control. Journal of Consulting and Clinical Psychol- personality traits on the success of a multimodal obesity
ogy, 37, 228–234. http://dx.doi.org/10.1037/h0031974 treatment. Behavioral Medicine, 37, 119–124. http://dx.doi.
Grant, B. F., Hasin, D. S., Stinson, F. S., Dawson, D. A., June org/10.1080/08964289.2011.635169
Ruan, W., Goldstein, R. B., . . . Huang, B. (2005). Prevalence, Leal, C., Bean, K., Thomas, F., & Chaix, B. (2012).
correlates, co-morbidity, and comparative disability of Multicollinearity in associations between multiple environ-
DSM-IV generalized anxiety disorder in the USA: Results mental features and body weight and abdominal fat: Using
from the National Epidemiologic Survey on Alcohol and matching techniques to assess whether the associations are
Related Conditions. Psychological Medicine, 35, 1747–1759. separable. American Journal of Epidemiology, 175,
http://dx.doi.org/10.1017/S0033291705006069 1152–1162. http://dx.doi.org/10.1093/aje/kwr434
Gurtman, M. B. (1991). Evaluating the interpersonalness of Lorentzen, S., & Høglend, P. (2004). Predictors of change
personality scales. Personality and Social Psychology Bulletin, during long-term analytic group psychotherapy. Psycho-
17, 670–677. http://dx.doi.org/10.1177/0146167291176010 therapy and Psychosomatics, 73, 25–35. http://dx.doi.org/
Gurtman, M. B. (2009). Exploring personality with the 10.1159/000074437
interpersonal circumplex. Social and Personality Psychology Ma, J., Thabane, L., Beyene, J., & Raina, P. (2016). Power
Compass, 3, 601–619. http://dx.doi.org/10.1111/j.1751- analysis for population-based longitudinal studies investi-
9004.2009.00172.x gating gene-environment interactions in chronic diseases: A
Hamilton, M. (1959). The assessment of anxiety states by simulation study. PLoS ONE, 11, e0149940. http://dx.doi.
rating. British Journal of Medical Psychology, 32, 50–55. org/10.1371/journal.pone.0149940
http://dx.doi.org/10.1111/j.2044-8341.1959.tb00467.x McEvoy, P. M., Burgess, M. M., & Nathan, P. (2013). The
Hardy, G. E., Cahill, J., Shapiro, D. A., Barkham, M., Rees, A., relationship between interpersonal problems, negative
& Macaskill, N. (2001). Client interpersonal and cognitive cognitions, and outcomes from cognitive behavioral group
styles as predictors of response to time-limited cognitive therapy for depression. Journal of Affective Disorders, 150,
therapy for depression. Journal of Consulting and Clinical 266–275. http://dx.doi.org/10.1016/j.jad.2013.04.005
Psychology, 69, 841–845. http://dx.doi.org/10.1037/0022- McEvoy, P. M., Burgess, M. M., & Nathan, P. (2014). The
006X.69.5.841 relationship between interpersonal problems, therapeutic
Hebert, E. A., Dugas, M. J., Tulloch, T. G., & Holowka, D. W. alliance, and outcomes following group and individual
(2014). Positive beliefs about worry: A psychometric cognitive behaviour therapy. Journal of Affective Disorders,
evaluation of the Why Worry-II. Personality and Individual 157, 25–32. http://dx.doi.org/10.1016/j.jad.2013.12.038
Differences, 56, 3–8. http://dx.doi.org/10.1016/j.paid.2013. Meyer, T. J., Miller, M. L., Metzger, R. L., & Borkovec, T. D.
08.009 (1990). Development and validation of the Penn State Worry
Hilbert, A., Saelens, B. E., Stein, R. I., Mockus, D. S., Welch, Questionnaire. Behaviour Research and Therapy, 28,
R. R., Matt, G. E., & Wilfley, D. E. (2007). Pretreatment 487–495. http://dx.doi.org/10.1016/0005-7967(90)90135-6
and process predictors of outcome in interpersonal and Molina, S., & Borkovec, T. D. (1994). The Penn State Worry
cognitive behavioral psychotherapy for binge eating disorder. Questionnaire: Psychometric properties and associated
Journal of Consulting and Clinical Psychology, 75, 645–651. characteristics. In G. C. L. Davey & F. Tallis (Eds.),
http://dx.doi.org/10.1037/0022-006X.75.4.645 Worrying: Perspectives on theory, assessment and treatment
Horowitz, L. M., Inouye, D., & Siegelman, E. Y. (1979). On (pp. 265–283). Oxford, England: Wiley.
averaging judges’ ratings to increase their correlation with Moras, K., di Nardo, P. A., & Barlow, D. H. (1992).
an external criterion. Journal of Consulting and Clinical Distinguishing anxiety and depression: Reexamination of
Psychology, 47, 453–458. http://dx.doi.org/10.1037/0022- the reconstructed Hamilton scales. Psychological Assessment,
006x.47.3.453 4, 224–227. http://dx.doi.org/10.1037/1040-3590.4.2.224
Horowitz, L. M., Rosenberg, S. E., Baer, B. A., Ureno, G., & Muthén, L. K., & Muthén, B. O. (2002). How to use a Monte
Vilasenor, V. S. (1988). Inventory of interpersonal prob- Carlo study to decide on sample size and determine power.
lems: Psychometric properties and clinical applications. Structural Equation Modeling, 9, 599–620. http://dx.doi.
Journal of Consulting and Clinical Psychology, 56, org/10.1207/S15328007SEM0904_8
885–892. http://dx.doi.org/10.1037//0022-006X.56.6.885 Newman, M. G., Castonguay, L. G., Borkovec, T. D., Fisher,
Jones, E. E., & Pulos, S. M. (1993). Comparing the pro- A. J., Boswell, J., Szkodny, L. E., & Nordberg, S. S. (2011).
cess in psychodynamic and cognitive-behavioral therapies. A randomized controlled trial of cognitive-behavioral
Journal of Consulting and Clinical Psychology, 61, 306–316. therapy for generalized anxiety disorder with integrated
http://dx.doi.org/10.1037/0022-006X.61.2.306 techniques from emotion-focused and interpersonal therapies.
Kachin, K. E., Newman, M. G., & Pincus, A. L. (2001). An Journal of Consulting and Clinical Psychology, 79, 171–181.
interpersonal problem approach to the division of social phobia http://dx.doi.org/10.1037/a0022489
subtypes. Behavior Therapy, 32, 479–501. http://dx.doi.org/ Newman, M. G., Castonguay, L. G., Jacobson, N. C., & Moore,
10.1016/S0005-7894(01)80032-0 G. A. (2015). Adult attachment as a moderator of treatment
Please cite this article as: Michelle G. Newman, et al., Interpersonal Problems Predict Differential Response to Cognitive Versus Behavioral
Treatment in a Randomized Controlled Trial, Behavior Therapy (2016), http://dx.doi.org/10.1016/j.beth.2016.05.005
interpersonal problems predict differential response 13
outcome for generalized anxiety disorder: Comparison Simon, G. E., & Perlis, R. H. (2010). Personalized medicine for
between cognitive–behavioral therapy (CBT) plus supportive depression: Can we match patients with treatments? American
listening and CBT plus interpersonal and emotional process- Journal of Psychiatry, 167, 1445–1455. http://dx.doi.org/
ing therapy. Journal of Consulting and Clinical Psychology, 10.1176/appi.ajp.2010.09111680
83, 915–925. http://dx.doi.org/10.1037/a0039359 Spielberger, C. D., Gorsuch, R. L., Lushene, R., Vagg, P. R., &
Newman, M. G., & Erickson, T. M. (2010). Generalized anxiety Jacobs, G. A. (1983). Manual for the State-Trait Anxiety
disorder. In J. G. Beck (Ed.), Interpersonal processes in the Inventory STAI (Form Y). Palo Alto, CA: Mind Garden.
anxiety disorders: Implications for understanding psychopa- Torgersen, S. (1986). Childhood and family characteristics in
thology and treatment (pp. 235–259). Washington, DC: panic and generalized anxiety disorders. American Journal
American Psychological Association. http://dx.doi.org/10. of Psychiatry, 143, 630–632. http://dx.doi.org/10.1176/ajp.
1037/12084-009 143.5.630
Newman, M. G., & Fisher, A. J. (2013). Mediated moderation in Vinnars, B., Barber, J. P., Norén, K., Thormählen, B., Gallop,
combined cognitive behavioral therapy versus component R., Lindgren, A., & Weinryb, R. M. (2007). Who can
treatments for generalized anxiety disorder. Journal of benefit from time-limited dynamic psychotherapy? A study
Consulting and Clinical Psychology, 81, 405–414. http://dx. of psychiatric outpatients with personality disorders.
doi.org/10.1037/a0031690 Clinical Psychology & Psychotherapy, 14, 198–210.
Ogrodniczuk, J. S., Piper, W. E., & Joyce, A. S. (2006). Treatment http://dx.doi.org/10.1002/cpp.530
compliance among patients with personality disorders receiv- Vittengl, J. R., Clark, L. A., & Jarrett, R. B. (2003).
ing group psychotherapy: What are the roles of interpersonal Interpersonal problems, personality pathology, and social
distress and cohesion? Psychiatry: Interpersonal and Biological adjustment after cognitive therapy for depression. Psycho-
Processes, 69, 249–261. http://dx.doi.org/10.1521/psyc.2006. logical Assessment, 15, 29–40. http://dx.doi.org/10.1037/
69.3.249 1040-3590.15.1.29
Paul, G. L. (1967). Strategy of outcome research in psycho- Welsh, G. S. (1952). An anxiety index and an internalization
therapy. Journal of Consulting Psychology, 31, 109–118. ratio for the MMPI. Journal of Consulting Psychology, 16,
http://dx.doi.org/10.1037/h0024436 65–72. http://dx.doi.org/10.1037/h0063398
Przeworski, A., Newman, M. G., Pincus, A. L., Kasoff, M. B., Wiggins, J. S., & Broughton, R. (1991). A geometric taxonomy
Yamasaki, A. S., Castonguay, L. G., & Berlin, K. S. (2011). of personality scales. European Journal of Personality, 5,
Interpersonal pathoplasticity in individuals with generalized 343–365. http://dx.doi.org/10.1002/per.2410050503
anxiety disorder. Journal of Abnormal Psychology, 120, Wolf, F. M. (1986). Meta-analysis: Quantitative methods for
286–298. http://dx.doi.org/10.1037/a0023334 research synthesis (Vol. 59). Newbury Park, CA: Sage
Renner, F., Jarrett, R. B., Vittengl, J. R., Barrett, M. S., Clark, L. A., Publications.
& Thase, M. E. (2012). Interpersonal problems as predictors of Wright, A. G., Pincus, A. L., Conroy, D. E., & Hilsenroth, M. J.
therapeutic alliance and symptom improvement in cognitive (2009). Integrating methods to optimize circumplex descrip-
therapy for depression. Journal of Affective Disorders, 138, tion and comparison of groups. Journal of Personality
458–467. http://dx.doi.org/10.1016/j.jad.2011.12.044 Assessment, 91, 311–322. http://dx.doi.org/10.1080/
Rosario-Martinez, H. (2013). phia: post-hoc interaction analysis. 00223890902935696
R package version 0.1–3.
Salzer, S., Pincus, A. L., Hoyer, J., Kreische, R., Leichsenring, F.,
& Leibing, E. (2008). Interpersonal subtypes within general- R E C E I V E D : January 9, 2016
ized anxiety disorder. Journal of Personality Assessment, 90, A C C E P T E D : May 26, 2016
292–299. http://dx.doi.org/10.1080/00223890701885076 A V A I L A B L E O N L I N E : xxxx
Please cite this article as: Michelle G. Newman, et al., Interpersonal Problems Predict Differential Response to Cognitive Versus Behavioral
Treatment in a Randomized Controlled Trial, Behavior Therapy (2016), http://dx.doi.org/10.1016/j.beth.2016.05.005