Received: 27 August 2016 Accepted: 20 September 2016

DOI: 10.1111/jebm.12236

ARTICLE

Exercise and sleep: a systematic review of previous

meta-analyses

George A Kelley Kristi Sharpe Kelley

Department of Biostatistics, Robert C.

Byrd Health Sciences Center, West Virginia Abstract
University, Morgantown, WV, USA Objective: Conduct a systematic review of previous meta-analyses on exercise and sleep out-
Correspondence comes in adults and a meta-analysis of studies nested within these meta-analyses.
George A Kelley, Department of Biostatistics,
Robert C. Byrd Health Sciences Center, West Methods: Meta-analyses of randomized controlled exercise interventions were included by
Virginia University, PO Box 9190, One Medical searching nine electronic databases and cross-referencing. Dual-selection and data abstraction
Center Drive, Morgantown, WV 26506–9190,
USA.
were conducted. Methodological quality of meta-analyses was assessed using AMSTAR and qual-
Tel: 304-293-6279; Fax: 304-293-5891; ity of evidence using GRADE. Random-effects models were used to pool results from the individ-
Email: [email protected] ual studies included in each meta-analysis.

Results: Three meta-analyses representing 950 adults were included. Methodological quality
ranged from 36% to 64% while quality of evidence was very low to low. Statistically significant
improvements (P ≤ 0.05) were observed for the apnea-hypopnea index (AHI), overall sleep qual-
ity, global score, subjective sleep, and sleep latency. The number-needed-to-treat (NNT) and per-
centile improvements ranged from 4 to 7 and from 18.1 to 26.5, respectively. When overall sleep
quality results from individual studies nested within different meta-analyses were pooled, statis-
tically significant standardized mean difference (SMD) improvements were observed (–0.50, 95%
CI –0.72 to –0.28). The NNT and percentile improvement were 7 and 19, respectively.

Conclusions: Exercise improves selected sleep outcomes in adults. To increase public health reach,
a large, well-designed, and more inclusive meta-analysis is needed.

KEYWORDS
apnea, exercise, meta-analysis, sleep, systematic review

1 INTRODUCTION While pharmacologic interventions are a common treatment for

sleep disorders,3,4 statistically significant adverse events have been
Sleep disorders are considered to be a major public health epidemic in reported, including an increased risk for falls and cognitive impairment
the United States, affecting an estimated 50 to 70 million US adults.1 among older adults.3 In addition, the magnitude of benefit has not been
These disorders have been associated with motor vehicle crashes and firmly established.3,4 Exercise, a low-cost, nonpharmacologic interven-
industrial disasters as well as medical and other occupational errors.1 tion that is readily available to the vast majority of adults, offers a
In addition, sleep disorders in adults have been associated with an potential complementary or alternative approach for improving sleep
increased risk for chronic diseases that include hypertension, type 2 and is particularly appealing in a public health setting.5
diabetes, depression, obesity and cancer.1 Furthermore, adults with Currently, systematic reviews with meta-analysis are considered by
sleep disorders report a lower quality of life and are less productive many to be the gold standard for determining the effects of an inter-
than those without a sleep disorder.1 Most notably, sleep disordered vention on an outcome.6,7 However, given that multiple systematic
adults are at an increased risk for all-cause mortality.1 In terms of reviews with meta-analysis now exist on the same topic,8 it becomes
economics, the total costs associated with sleep disorders have been difficult to make evidence-based decisions regarding the intervention
estimated to be as high as $166 billion per year in the United effects on the outcome of interest in the population of interest. There-
States.2 fore, it is now necessary to systematically study previous systematic


c 2016 Chinese Cochrane Center, West China Hospital of Sichuan University and John Wiley & Sons Australia, Ltd

J Evid Based Med. 2017; 10: 26–36 wileyonlinelibrary.com/journal/jebm 26

GA KELLEY AND KS KELLEY 27

reviews of meta-analyses in order to provide healthcare personnel and propriate intervention (e.g., pharmacological trial), (c) inappropriate
policy-makers with the information they need to make better decisions comparison (e.g., aerobic exercise versus drug, analyzing the difference
regarding the effectiveness of an intervention on the outcome of inter- in the exercise group while not accounting for the control group, etc.),
est, provide researchers with information to inform future original (d) inappropriate outcome (e.g., anxiety), and (e) inappropriate study
studies, and provide meta-analysts with information to inform future type (e.g., systematic review with no meta-analysis included, meta-
meta-analyses, including whether an updated meta-analysis should be analysis that did not report data separately for randomized controlled
conducted on the topic of interest. trials only, etc.).
Previous meta-analyses have reached conflicting conclusions
regarding the effects of exercise on sleep.9–17 In addition, to the best
2.2 Data sources
of the authors’ knowledge, no previous systematic review of sys-
tematic reviews with meta-analysis examining the effects of exercise Potentially eligible studies were derived by electronic database
on sleep in adults has been performed. Therefore, given multiple searches, cross-referencing from retrieved articles and inspection of
systematic reviews with meta-analysis on exercise and sleep as well as the first author’s files. For electronic searching, nine databases were
the conflicting findings of such,9–17 the need to systematically review searched from their inception up to June 14, 2015 using the graph-
multiple meta-analyses for both applied and research purposes,8 ical user interface for each database. Databases searched included
and the absence of any previous systematic review of systematic PubMed, Sport Discus, Web of Science, Scopus, PsychInfo, Cochrane
reviews with meta-analysis of randomized controlled trials on this Database of Systematic Reviews, Physiotherapy Evidence Database
topic, the objectives of this study were to conduct a systematic (PEDro), Database of Abstract of Reviews of Effects (DARE), and Pro-
review of previous meta-analyses on exercise and sleep outcomes quest. Scopus was included because it has been reported to pro-
in adult humans, and conduct a meta-analysis on exercise and vide coverage of EMBASE, a database that was not accessible to
sleep outcomes based on the individual studies nested within these the investigators.21 Keywords or forms of keywords used in the
meta-analyses. database searches included exercise, physical fitness, randomized, sys-
tematic review, meta-analysis, sleep, apnea, and insomnia. A copy of
the search strategies used for each database is shown in Supplemen-
tary File 1. Following duplicate removal both electronically and man-
2 METHODS
ually, the overall precision of the searches was calculated by divid-
ing the number of studies that met the eligibility criteria by the
2.1 Study eligibility
total number of studies screened after removing duplicates.22 The
This systematic review of previous systematic reviews with meta- number needed to read (NNR) was then calculated as the recipro-
analysis is registered in the International Prospective Register of Sys- cal of the precision.22 All studies were stored in Reference Manager,
tematic Reviews (PROSPERO) trial registry (CRD42015023449). In version 12.0.23
addition and where applicable, the general guidelines of the Preferred
Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA)
2.3 Study selection
Statement were followed.18
The inclusion criteria for this study, established a priori, were Both authors selected all studies independent of each other. They then
as follows: (1) previous systematic reviews with meta-analysis of convened and reviewed their selections for concurrence. Any incon-
randomized controlled trials or data reported separately for ran- gruities were resolved by consensus.
domized controlled trials if the meta-analysis included other study
designs, (2) adult humans ≥18 years of age, (3) exercise (aerobic,
2.4 Data abstraction
strength or both) as the intervention, (4) published and unpub-
lished (dissertations and master’s theses) studies in any language Prior to coding studies, coding sheets were developed using Microsoft
up through June of 2015, and (5) exercise minus control group dif- Excel (2010).24 The coding sheets could hold up to 284 items from
ferences in sleep as an outcome in the original meta-analysis and each included meta-analysis. The major categories of variables coded
reported as the standardized mean difference (SMD) effect size or included (a) study characteristics (source, year, impact factor of jour-
calculable using the SMD if at least two studies were pooled. The focus nal, etc.), (b) participant characteristics (age, gender, condition(s), etc.),
on limiting meta-analyses to randomized controlled trials is based (c) intervention characteristics (length, frequency, intensity, duration,
on previous research suggesting that they are the only way to con- type of exercise, compliance, etc.), and (d) data for sleep outcomes
trol for unknown confounders and that nonrandomized controlled (sample sizes, means, variances, etc.) at both the pooled meta-analytic
trials tend to overestimate the effects of treatment(s) in healthcare level as well as for each study included in each meta-analysis. All data
interventions.19,20 The SMD was selected as the metric of choice given were coded by both authors, independent of each other. After coding
the different instruments used to assess selected sleep outcomes as was completed, all items were reviewed by both authors for correct-
well as the desire to compare results using the same metric. Broadly, ness. Any inconsistencies were resolved by consensus. Using Cohen’s
studies were excluded based on at least one of the following: (a) kappa statistic,25 the overall agreement rate prior to correcting any dif-
inappropriate population (e.g., children and/or adolescents), (b) inap- ferences was kappa = 0.96, considered to be “excellent”.26
28 GA KELLEY AND KS KELLEY

2.5 Methodological quality were computed if the findings were statistically significant and the
requisite data from each study included in each meta-analysis were
Methodological quality of each meta-analysis was assessed using
provided.38–40 Prediction intervals are used to approximate the treat-
The Assessment of Multiple Systematic Reviews (AMSTAR)
ment effect in a new study38–40 and may be more applicable for deci-
Instrument.27–30 This instrument was chosen because of (a) its
sion analysis.41
construct validity (intra-class correlation coefficient = 0.84), (b)
To enhance practical application, the number-needed-to-treat
inter-rater reliability (kappa = 0.70), and (c) feasibility (average of
(NNT) was estimated for any summary findings that were reported as
15 minutes per study to complete).29 Responses of “Yes,” “No,” “Can’t
statistically significant. This was accomplished using a control group
Answer,” or “Not Applicable” are possible for this 11-item question-
risk of 30%.6 Additionally, Cohen’s U3 index was calculated to esti-
naire. The “Can’t Answer” response is chosen when an item is not
mate the percentile gain in the intervention group.42 Results for
described but relevant while the “Not Applicable” response is chosen
small-study effects (publication bias, etc.) were also abstracted or
when an item is not relevant (e.g., assessment of publication bias not
calculated using the regression-intercept approach of Egger et al.,43
possible because of a lack of studies).27–30 For consistency with the
assuming that adequate information were available and the number
other questions, the question “Was the status of publication (i.e., grey
of SMDs was ≥10.44 One-tailed alpha values ≤0.05 for the inter-
literature) used as an inclusion criterion?” was changed to “Was the
cept were considered to represent statistically significant small-study
status of publication (i.e., grey literature) as an inclusion criterion
effects.
avoided?” Both authors evaluated the methodological quality of each
study independent of each other. They then met and reviewed every
rating for agreement. Disagreements were resolved by consensus. The
overall agreement rate prior to correcting discrepancies was kappa =
0.58, considered to be “good”.26 2.6.2 Meta-analysis based on studies nested within
In addition to AMSTAR, the overall quality of the evidence was included meta-analyses
assessed using the Grades of Recommendations Assessment, Devel-
To increase generalizability, the investigators also conducted their own
opment and Evaluation (GRADE) instrument.31 Overall quality was
meta-analysis based on available sleep outcome results from the indi-
classified as either very low, low, moderate, or high.31 To assess
vidual studies nested within each included meta-analysis and while
impact, the total number of times that each included meta-analysis
avoiding duplication, that is, results of the same study reported in two
was cited as well as the average number of citations per year
or more different meta-analyses. Data synthesis included the abstrac-
was calculated. This was accomplished using version 4.17 of Pub-
tion and pooling of results (sample sizes, SMD, variance statistics,
lish or Perish (Google Scholar Citation mechanism)32 on August 5,
etc.) from each study included in each meta-analysis into one over-
2015.
all finding for similar outcomes (e.g., overall sleep quality). All anal-
yses were limited to data reported in the retrieved meta-analyses

2.6 Data synthesis because the focus of the current investigation was on each meta-
analysis and not the original studies. Pooling of studies was accom-
2.6.1 Summary findings for sleep outcomes from each plished using a random-effects, method-of-moments model.45 Hetero-
meta-analysis geneity was assessed using the Q statistic36 and inconsistency using
The summary findings from each meta-analysis were extracted with a I2 .37 A two-tailed, z-based alpha value ≤0.05 for the SMD was con-
focus on random versus fixed-effect models because the former incor- sidered statistically significant. In addition, 95% CI were calculated.
porates between-study heterogeneity into the analysis when pooling Based on recent recommendations, small-study effects were exam-
results.33,34 The SMD was the primary metric of interest along with ined using funnel plots and Egger’s regression intercept test.44 A one-
its 95% confidence intervals (CI), z statistic and alpha value. If suffi- tailed alpha value ≤ 0.05 for the intercept was considered to repre-
cient data were available or it was feasible, these data were calcu- sent statistically significant small-study effects. Outliers were consid-
lated if not reported in the original study. The magnitude of effect for ered to be those in which the alpha values for the standardized resid-
each SMD from each meta-analysis was categorized as either trivial uals were ≤0.05. In addition, influence analysis was conducted with
(<0.20), small (0.20 to 0.49), medium (0.50 to 0.79) or large (≥0.80).35 each study deleted from the model once as well as cumulative meta-
Two-tailed alpha levels ≤0.05 for z were considered statistically signif- analysis, ranked by year. Furthermore, 95% PI, NNT based on a con-
icant. In addition, Q, a measure of heterogeneity, was also extracted trol group risk of 30% and percentile improvement using Cohen’s U3
or calculated for each outcome if data were available to do so.36 An index were calculated. Finally, simple random-effects meta-regression
alpha value ≤0.10 was considered to represent statistically significant (method-of-moments approach) was used to examine the association
heterogeneity.37 The I2 statistic, a gauge of inconsistency, was also between changes in sleep and the meta-analysis from which the results
extracted or calculated if appropriate data were provided.37 Values were derived.45 A two-tailed, z-based alpha value ≤0.05 for the slope
of I2 were categorized as either low (0% to <25%), moderate (25% to (𝛽 1 ) was considered statistically significant. Negative SMD’s were con-
<50%), large (50% to <75%) or very large (≥75%).37 sidered to represent improvements in sleep. All analyses were carried
It was assumed, a priori, that none of the included meta-analyses out using Comprehensive Meta-Analysis (version 3.3)46 and Microsoft
would include prediction intervals (PI).38–40 Consequently, 95% PI Excel 2010.24
GA KELLEY AND KS KELLEY 29

FIGURE 1 Flow diagram for selection of articles

3 RESULTS One meta-analysis was limited to participants with obstructive sleep

apnea.15 Length of training for the studies included in each meta-
3.1 Characteristics of included meta-analyses analysis ranged from 5 to 52 weeks, frequency from 3 to 10 times
per week, and duration from 20 to 90 minutes per session.9,14,15
A total of 392 references were initially identified. After removing
The one meta-analysis that provided information on adverse events
duplicates both electronically and manually, 283 (72.2%) remained. Of
reported that six studies did not provide any information while one
the 283 independent citations screened, three aggregate data meta-
study reported that adverse events were minimal.9 Both supervised
analyses met all eligibility criteria.9,14,15 Search precision after remov-
and unsupervised exercise were performed and included both aer-
ing duplicates was 0.01, while the NNR was 94. A flow diagram that
obic and/or strength training.9,14,15 For the two meta-analyses that
describes the search process is shown in Figure 1 while a list of
reported data,9,14 intensity of training was classified as moderate to
excluded studies, including the reasons for exclusion, can be found in
vigorous. For the one meta-analysis that reported information, compli-
Supplementary File 2. Studies were excluded based on an inappro-
ance, defined as the percentage of exercise sessions attended, ranged
priate study design (41.4%) as well as an inappropriate intervention
from 32.4% to 93.3% for the studies included in their review.9 One
(36.8%), outcome (17.9%), population (3.6%), and comparison (0.4%).
meta-analysis only included studies in which walking was part of
Table 1 describes the general characteristics of each meta-analysis. As
the exercise intervention.9 Sleep outcomes were assessed using the
can be seen, the three included studies were published between 2013
apnea-hypopnea index (AHI)15 and Pittsburgh Sleep Quality Index
and 2015, included 2 to 9 studies and between 63 and 599 men and
(PSQI).14 Another meta-analysis reported the assessment of overall
women (total N = 950).9,14,15 One study reported receiving govern-
sleep quality but specific measures for the assessment of such from
ment funding for their work.9 All three meta-analyses included differ-
each of the included studies were not provided.9
ent types of populations9,14,15 in adults up to 72 years of age.9,14,15
30 GA KELLEY AND KS KELLEY

TA B L E 1 General characteristics of included meta-analyses

Study Year Country Studies Participants Interventions Sleep assessment

15 a
Araghi et al. 2013 UK 2 63 men and women with obstructive sleep Exercise lasting 12 weeks AHI
apnea, 46–54 years of age ( X = 50)
Chiu et al.9 2015 China 9 599a men and women with cancer (42% breast), Supervised and unsupervised –
≥18 years of age (X ± SD, 54.4 ± 5.7) walking included in
interventions lasting 5–
35 weeks (X = 10), frequency
3–10× week (X = 4.5),
duration 20–90 minutes per
session ( X = 37.5), intensity
classified as moderate in 7
studies, moderate to vigorous
in 1, and not reported in
another, compliance
32.4–93.3% ( X = 77.0%)
Yang et al.14 2012 China 6 305 older adults (171 exercise, 134 control) Aerobic and/or strength training PSQI (global score,
with sleep problems, 48.6–72 years of age 10–52 weeks (X ± SD, 21 ± subscales of
(X ± SD, 62.7 ± 7.4) 15), frequency 3–5× week (X ± subjective sleep,
SD, 4 ± 1), duration 10–60 sleep latency, sleep
minutes per session, intensity duration, sleep
60–75% HRR (1 study), efficiency, sleep
60–85% HRR (1 study), disturbance, daytime
60–85% MHR (1 study), functioning)
55–75% MHR (1 study)

Note: X ± SD, mean ± standard deviation. Description of meta-analyses limited to those studies nested within each meta-analysis that met all eligibility
criteria for this study. Data presented limited to what was reported or could be calculated from reported data. Number of participants limited to those in
which results were calculated. –, data not provided or insufficient data to calculate; HRR, heart rate reserve; MHR, maximum heart rate; PSQI, Pittsburgh
Sleep Quality Index; AHI, Apnea-hypopnea Index.
a
Separate sample sizes not available for exercise and control groups.

3.2 Methodological quality and impact No statistically significant differences were observed for sleep dura-
tion, efficiency, disturbance, or daytime functioning.14
The results for each meta-analysis using the AMSTAR instrument are
For those results that were statistically significant, no statistically
shown in Supplementary File 3. As can be seen, the overall quality of
significant heterogeneity or inconsistencies were observed for the
the meta-analyses using the AMSTAR instrument ranged from 36% to
AHI.15 However, statistically significant heterogeneity and a large
64%.9,14,15 All of the studies were considered to have provided ade-
amount of inconsistency were observed for overall sleep quality in
quate information regarding (a) an a priori design, (b) description of
both meta-analyses that assessed such9,14 as well as subjective sleep,
study characteristics, (c) assessment of study quality, and (d) appropri-
sleep latency, sleep duration, sleep efficiency and sleep disturbance in
ate methods for combining studies.9,14,15 In contrast, none of the stud-
the one study that reported this information.14 Nonoverlapping pre-
ies provided a reference list of excluded studies, including the reasons
diction intervals were observed for overall sleep quality in the study by
for exclusion, as well as appropriate information regarding conflicts of
Chiu et al.9 but not Yang et al.14 Overlapping prediction intervals were
interest, especially with respect to conflicts of interest from each of the
also observed for subjective sleep as well as sleep latency.14 None of
studies included in the meta-analyses.9,14,15 The results for the other
the studies reported results for potential small-study effects (publica-
five criteria were mixed.9,14,15
tion bias, etc.).9,14,15
In relation to impact, the total number of times that each meta-
The NNT and percentile improvement estimates for statistically sig-
analysis was cited were 0,9 24,15 and 56.14 When adjusted for the num-
nificant findings are shown in Table 3. Assuming a control group risk
ber of years that each meta-analysis was available, citation rates were
of 30%, the NNT for sleep outcomes ranged from a low of 4 for sleep
0,9 12,15 and 18.7.14
latency to a high of 7 for overall sleep quality.14 Overall percentile
improvements were similar, ranging from 18.1 to 26.5. Correspond-
ing 95% CI were best for sleep quality in the study by Chiu et al.,9 and
3.3 Data synthesis worst for sleep latency in the study by Yang et al.14 Using the GRADE

3.3.1 Results from each meta-analysis instrument, the overall quality of evidence ranged from very low to low
(Supplementary File 4).
Overall results for the three included meta-analyses are shown in
Table 2.9,14,15 As can be seen, the number of ES’s in each meta-analysis
were small, ranging from 2 to 9, while the number of participants 3.3.2 Results of pooling different studies from different
ranged from 63 to 599.9,14,15 Statistically significant improvements meta-analyses for the same outcomes
were observed for all three meta-analyses.9,14,15 These included the Based on the availability of evidence, the pooling of studies from
AHI,15 overall sleep quality,9,14 subjective sleep14 and sleep latency.14 each meta-analysis was limited to overall sleep quality from two of
GA KELLEY AND KS KELLEY 31

TA B L E 2 Overall posttreatment changes in sleep from included meta-analyses

Study ES/participants (No.) X (95% CI) Z (p) Q (p) I2 (%) Tau2 PI (95%)
15
Araghi et al. (2013)
AHI 2/63 –0.72 (–1.23, –0.21) 2.79 (0.005)a 0.005 (0.94) 0 <0.001 –
Chiu et al. (2015)9
Overall sleep quality 9/599 –0.52 (–0.79, –0.25) 3.77 (0.0002)a 20.3 (0.009)a 61 0.019 –0.98, –0.06
14
Yang et al. (2012)
Global score 5/288 –0.47 (–0.86, –0.08) 2.35 (0.02)a 9.99 (0.04)a 60 0.12 –1.74, 0.80
Subjective sleep 5/239 –0.47 (–0.73, –0.20) 3.47 (0.0005)a 9.06 (0.06) 56 0.13 –1.70, 0.76
Sleep latency 5/239 –0.58 (–1.08, –0.08) 2.26 (0.02)a 12.2 (0.02)a 67 0.21 –2.25, 1.09
Sleep duration 6/305 –0.10 (–0.53, 0.33) 0.47 (0.64) 15.6 (0.008)a 68 0.19 –
Sleep efficiency 5/239 –0.35 (–0.84, 0.15) 1.38 (0.17) 12.0 (0.02)a 67 0.20 –
Sleep disturbance 4/245 –0.25 (–0.96, 0.47) 0.68 (0.50) 20.4 (0.0001)a 85 0.44 –
Daytime functioning 5/262 –0.23 (–0.48, 0.02) 1.81 (0.07) 6.9 (0.14) 42 – –

Note: No., number; ES, effect size; X (95% CI), mean difference and 95% confidence intervals; Z (p), Z-value and probability value for Z; Q (p), Cochran’s Q
statistic and associated alpha (p) value for Q; I2 , I2 statistic for inconsistency; PI, prediction intervals; negative values represent improvements in sleep; –,
data not calculated or calculable; boldfaced values represent statistically significant changes.
a
Statistically significant (P ≤ 0.05).

TA B L E 3 NNT and percentile improvement for statistically signifi- quality and the meta-analysis from which results were derived (𝛽 1 =
cant sleep outcomes 0.05, 95% CI –0.41 to 0.52, P = 0.82). Overlapping prediction intervals
Study NNT (95% CI) U3 index (95% CI)a were observed (95% PI, –1.19 to 0.20). The NNT was 7 (95% CI 5 to 10),
Araghi et al. (2013)15 5 (4, 13) 26.5 (8.5, 39) while the percentile improvement was 19 (95% CI 11.2 to 26.1).
AHI
Chiu et al. (2015)9 4 DISCUSSION
Overall sleep quality 6 (5, 12) 19.8 (9.9, 28.5)
Yang et al. (2012)14 4.1 Findings
Global score (PSQI) 7 (5, 34) 18.1 (3.2, 30.5)
In the ideal scenario supporting the effects of any intervention on an
Subjective sleep (PSQI) 5 (7, 14) 18.1 (7.9, 26.7)
outcome, results will be (a) statistically significant with nonoverlapping
Sleep latency (PSQI) 4 (6, 34) 21.9 (2.8, 36.0)
and narrow CI, (b) homogeneous with no inconsistency, (c) have nar-
Note: NNT, number needed to-treat, calculated from SMD and 95% confi- row and nonoverlapping PI, (d) be free of all bias, including small-study
dence intervals and assuming a control group risk of 30%; 95% CI, 95% con-
effects, and (e) be stable with a magnitude of change that is practi-
fidence intervals.
a
Cohen’s U3 index for percentile improvement; AHI, apnea-hypopnea cally important. However, satisfying all these criteria is probably highly
index; PSQI, Pittsburgh Sleep Quality Index. unlikely.
Rather, one must draw inferences based on imperfect findings. Such
is the case with the current investigation. From the investigative team’s
the included meta-analyses.9,14 This included 14 different studies rep- perspective, the overall findings of the current study suggest that exer-
resenting 887 participants.9,14 As shown in Figure 2, a moderate cise improves selected sleep outcomes in the sample of adults included.
and statistically significant SMD improvement in overall sleep quality This interpretation is reinforced by (a) statistically significant improve-
was observed (z = 4.6, P < 0.001) along with statistically significant ments in the AHI, overall sleep quality, subjective sleep, and sleep
heterogeneity (Q = 30.7, P = 0.004) and a large amount of inconsis- latency, (b) low NNT (4 to 7) for statistically significant outcomes, (c)
tency (I2 = 57.7%, 95% CI 23.4% to 76.6%). percentile improvements (18.1 to 26.5) for statistically significant out-
No statistically significant outliers were observed (P for all stan- comes, and (d) nonoverlapping PI for overall sleep quality in the meta-
dardized residuals > 0.05). Visual inspection of the funnel plot analysis by Chiu et al.9 In contrast, the potential benefits of exercise
(Fig. 3) as well as Egger’s regression intercept test suggests that sta- on sleep in adults may be questioned given (a) statistically significant
tistically significant small-study effects were present (𝛽 0 = –3.24, P = heterogeneity for three of the five statistically significant outcomes,
0.04). As can be seen in Figure 4, influence analysis revealed that SMD (b) a large amount of inconsistency for four of the five outcomes in
results were statistically significant with each study deleted from the which statistically significant improvements were observed, (c) over-
model once, with overall changes differing by 0.11 (20.8%). Cumulative lapping PI for global sleep score, subjective sleep and sleep latency,
meta-analysis demonstrated that improvements in sleep quality have (d) lack of statistically significant improvements for sleep duration,
remained statistically significant since the first study was published in sleep efficiency, sleep disturbance and daytime functioning, (e) based
1997 (Fig. 5). No association was observed for changes in overall sleep on AMSTAR assessment, a quality rating of less than 50% for two of the
32 GA KELLEY AND KS KELLEY

Group by Original Study Statistics for each study Point estimate and 95% CI
Meta-analysis
Point Lower Upper
estimate limit limit
Chiu et al., 2015 Cheville et al., 2013 -0.86 -1.40 -0.31
Chiu et al., 2015 Donnellyet al., 2011 -0.55 -1.22 0.13
Chiu et al., 2015 Mocket al., 1997 -0.67 -1.26 -0.09
Chiu et al., 2015 Rogers et al., 2014 -0.15 -0.75 0.45
Chiu et al., 2015 Sprod et al., 2010 -0.16 -0.79 0.46
Chiu et al., 2015 Tang et al., 2010 -1.23 -1.73 -0.73
Chiu et al., 2015 Wang et al., 2011 -0.79 -1.27 -0.32
Chiu et al., 2015 Wenzel et al., 2013 -0.21 -0.56 0.14
Chiu et al., 2015 Wiskemann et al., 2011 -0.13 -0.51 0.26
Chiu et al., 2015 -0.52 -0.79 -0.25
Yang et al., 2012 Elavsky& Mcauley, 2007 -0.15 -0.55 0.25
Yang et al., 2012 Irwin et al., 2008 -0.35 -0.91 0.21
Yang et al., 2012 King et al., 1997 -1.23 -1.89 -0.57
Yang et al., 2012 King et al., 2008 -0.12 -0.61 0.37
Yang et al., 2012 Singh et al., 1997 -0.81 -1.59 -0.04
Yang et al., 2012 -0.47 -0.86 -0.08
Overall -0.50 -0.72 -0.28
-2.00 -1.00 0.00 1.00 2.00
Favors Exercise Favors Control

F I G U R E 2 Forest plot for changes in overall sleep quality. The black horizontal lines represent the 95% confidence intervals while the squares
represent the point estimate. The first two black diamonds represent the overall point estimate and 95% confidence intervals from each meta-
analysis, while the third black diamond represents the overall pooled point estimate and 95% confidence intervals from all individual studies
included in each meta-analysis. All analyses are based on the random-effects model

Funnel Plot of Precision by Point estimate

6

4
Precision (1/Std Err)

-2.0 -1.5 -1.0 -0.5 0.0 0.5 1.0 1.5 2.0

Point estimate

FIGURE 3 Funnel plot for overall changes in sleep quality

three meta-analyses conducted,9,14 and (f) based on GRADE assess- percentile improvement,19 (d) stability of findings when each study
ment, low- to very low-quality levels of evidence for all outcomes. was deleted from the model once, (e) statistical significance of findings
The potential benefits of exercise on overall sleep quality may be since the conduct of the first study in 1997, and (f) no statistically sig-
especially promising given that the investigative team, based on the nificant association between the meta-analysis from which the results
results reported in the original meta-analyses, were able to success- were derived. However, these findings may have been weakened by (a)
fully combine the results for overall sleep quality from two of the statistically significant heterogeneity, (b) a large amount of inconsis-
included meta-analyses.9,14 These findings included (a) statistically sig- tency, (c) overlapping PI, (d) potential small-study effects, and (e) based
nificant improvements in overall sleep quality, (b) a low NNT,7 (c) a large on GRADE, the very low to low quality of evidence.
GA KELLEY AND KS KELLEY 33

Original Study Point estimate (95%

CI) with study removed
Lower Upper
Point limit limit
Tang et al., 2010 -0.42 -0.61 -0.23
King et al., 1997 -0.45 -0.65 -0.24
Cheville et al., 2013 -0.47 -0.69 -0.25
Wang et al., 2011 -0.47 -0.70 -0.25
Singh et al., 1997 -0.48 -0.70 -0.26
Mock et al., 1997 -0.49 -0.71 -0.26
Donnelly et al., 2011 -0.50 -0.72 -0.27
Irwin et al., 2008 -0.51 -0.74 -0.28
Sprod et al., 2010 -0.52 -0.75 -0.30
Rogers et al., 2014 -0.52 -0.75 -0.30
Wenzel et al., 2013 -0.53 -0.76 -0.30
King et al., 2008 -0.53 -0.75 -0.30
Elavsky & Mcauley, 2007 -0.53 -0.76 -0.31
Wiskemann et al., 2011 -0.53 -0.76 -0.31
-0.50 -0.71 -0.28
-1.00 -0.50 0.00 0.50 1.00
Favors Exercise Favors Control

F I G U R E 4 Influence analysis for changes in overall sleep quality with each study deleted from the model once. The black horizontal lines repre-
sent the 95% confidence intervals, while the squares represent the point estimate. The black diamond represents the overall point estimate and
95% confidence intervals

Original Study Cumulative statistics Cumulative point

estimate (95% CI)
Lower Upper
Point limit limit
Mock et al., 1997 -0.67 -1.26 -0.09
King et al., 1997 -0.93 -1.47 -0.39
Singh et al., 1997 -0.89 -1.27 -0.51
Elavsky & Mcauley, 2007 -0.67 -1.16 -0.17
Irwin et al., 2008 -0.59 -0.97 -0.20
King et al., 2008 -0.49 -0.83 -0.16
Sprod et al., 2010 -0.44 -0.74 -0.15
Tang et al., 2010 -0.57 -0.90 -0.23
Donnelly et al., 2011 -0.56 -0.86 -0.26
Wang et al., 2011 -0.59 -0.86 -0.31
Wiskemann et al., 2011 -0.53 -0.79 -0.28
Cheville et al., 2013 -0.56 -0.80 -0.32
Wenzel et al., 2013 -0.52 -0.75 -0.30
Rogers et al., 2014 -0.50 -0.71 -0.28
-0.50 -0.71 -0.28
-2.00 -1.00 0.00 1.00 2.00
Favors Exercise Favors Control

F I G U R E 5 Cumulative meta-analysis, ranked by year, for point estimate changes in overall sleep quality. The black diamond represents the overall
point estimate and 95% confidence intervals

The statistically significant findings observed in this study for long-term improvements up to three months (SMD, –0.29 95% CI –
selected outcomes are somewhat less than the use of pharmacological 0.52 to –0.06) were reported.48 The present findings also compare
interventions to improve sleep. For example, the approximate 19.7% favorably to a recent meta-analysis on meditative movement therapies
improvement in AHI in the meta-analysis by Araghi et al.,15 compares in adults >60 years of age and in which SMD improvements of –0.70
to improvements ranging from 25% to 45% with the use of various (95% CI –0.96 to –0.43) were reported.49 Thus, it appears that exercise
pharmacologic agents in participants with obstructive sleep apnea.47 may serve as a complementary or alternative approach for improving
However, the authors of this prior study concluded that there was sleep in adults.
insufficient evidence to recommend drug therapy in the treatment of
obstructive sleep apnea.47 In contrast, the current findings for over-
4.2 Implications for research
all sleep quality compare favorably to a recent meta-analysis of mind-
body interventions in cancer patients in which statistically signifi- There are at least six recommendations for future research using
cant SMD improvements of –0.43 (95% CI –0.24 to –0.62) as well as the meta-analytic approach to examine the effects of exercise on
34 GA KELLEY AND KS KELLEY

sleep outcomes in adults. First, the methodological quality of the adults, including what type of exercise, aerobic, strength training, or
meta-analyses themselves could be improved. This includes (a) the both, may be best for improving selected sleep outcomes. More accu-
inclusion of studies regardless of publication status or providing a rate data on this topic should lead to better treatment in the population
strong rationale for not doing so, (b) providing a bibliography of of interest. Third, future randomized controlled trials need to report
all excluded studies, including the reasons for exclusion, and (c) complete information on any adverse events experienced by the par-
providing a description of potential conflicts of interest, including ticipants during the intervention.
potential sources of support, for each of the studies included in each
meta-analysis.
4.3 Implications for practice
Second, based on citation rates, the impact of the included meta-
analyses appears to be small. One possible explanation may be that The findings of the current review provide important information for
this work is published in journals that do not have a large reader- practice. First, despite the low quality of evidence as well as lack
ship, thereby compromising the reach of this potentially beneficial of statistically significant results for several sleep outcomes, exercise
non-pharmacological intervention. A second possible reason may be appears to improve selective sleep outcomes, including more global
the fact that guidelines for the treatment of sleep problems such measures of sleep. While no specific recommendations directed solely
as insomnia are heavily focused on pharmacological versus non- at sleep outcomes can be made and further research is needed, it
pharmacological therapies.50 Another reason may be that the number would appear pragmatic to suggest that adherence to current and
of researchers and practitioners interested in sleep problems may be broad guidelines for exercise be recommended. These include at least
less than those interested in other conditions such as cancer and car- 150 minutes per week of moderate-intensity activity such as brisk
diovascular disease. walking or 75 minutes or more each week of vigorous-intensity activ-
Third, future meta-analyses should provide practical information so ity such as jogging.51 Some combination of the two is also acceptable.51
that practitioners and policy-makers can make better evidence-based Additionally, at least two days per week of muscle strengthening activ-
decisions with respect to the effects of exercise on sleep in adults. ities that exercise the major muscle groups of the body (legs, hips, back,
These include, but are not necessarily limited to, statistics such as rela- abdomen, chest, shoulders, and arms) should be performed.51 How-
tive changes, NNT and/or percentile improvements. ever, it is important to note that these are general recommendations.51
Fourth, future meta-analyses should include PIs as well as CIs. The
use of PIs can help to establish expected outcome effects in a new study
4.4 Strengths and potential limitations of this study
and may also be more valid for decision-making.40 However, it’s impor-
tant to understand that PIs are based on random mean effects while There are at least three strengths of the current study. First, to the
confidence intervals are not.40 best of the authors’ knowledge, this is the first systematic review of
Fifth, the three meta-analyses included in the current study were previous systematic reviews with meta-analysis directed at determin-
limited to less than 10 effect sizes for each outcome as well as partici- ing the effects of exercise on selected sleep outcomes in adults. This
pants with certain characteristics: obstructive sleep apnea,15 cancer,9 is important for (a) determining the effects of exercise on sleep out-
and older adults with sleep problems.14 Given the former, it would comes, (b) providing recommendations on the reporting and conduct
appear plausible to suggest that a larger more inclusive meta-analysis of future research, and (c) providing evidence regarding the priori-
would be a more powerful research design and have greater applicabil- tization of exercise over alternative treatments such as pharmaco-
ity across a wider range of participants, one of the very reasons for con- logic interventions.8 As a result, a summary of previous meta-analyses
ducting a meta-analysis. This approach may be particularly important if addressing the effects of exercise on sleep outcomes is now avail-
viewed from the perspective of increasing public health reach. In addi- able, thereby contributing important evidence for advancing future
tion, a larger and more inclusive meta-analysis would provide one with research, practice and policy-making. Second, the additional analyses
a greater opportunity to examine for potential predictors with respect that were conducted but not available in the original meta-analyses
to changes in selected sleep outcomes. (NNT, percentile improvement, PIs)9,14,15 aided in strengthening the
Sixth, future meta-analyses should report information on adverse evidence from which conclusions could be drawn from the included
events for all included studies, including whether the original studies studies. Importantly, the calculation of PIs provides future researchers
provided such information. This is important for balancing the benefits with valuable information in the planning and conduct of randomized
and harms of any intervention, including exercise. controlled studies aimed at determining the effects of exercise on sleep
Based on the current findings, three recommendations for future in adults. Third, to the best of the investigative team’s knowledge, this
randomized controlled trials appear to be warranted. First, a need is the first systematic review of previous systematic reviews with meta-
exists for the inclusion of data on the cost-effectiveness of exer- analysis in which a meta-analysis was conducted based on a similar out-
cise interventions on sleep outcomes in adults given that none come from different studies included in different meta-analyses.9,14
of the meta-analyses reported such data.9,14,15 This is of course Such an approach is a cost and time efficient way to increase statisti-
assuming that the original studies included in each of the meta- cal power for primary endpoints and enhance generalizability.
analyses did not provide this information. Second, a need exists for The current study may be subject to at least three possible limita-
multi-arm randomized controlled trials that directly compare the tions. First, the number of studies and subsequent effect sizes included
dose–response effects of exercise on selected sleep outcomes in in each meta-analysis was small and limited to very narrowly defined
GA KELLEY AND KS KELLEY 35

populations. As a result, the findings may not be generalizable to other 10. Mishra SI, Scherer RW, Geigle PM, Berlanstein DR, Topaloglu O,
populations. Despite this, it’s important to realize that two is the min- Gotay CC, et al. Exercise interventions on health-related quality of
life for cancer survivors. Cochrane Database of Systematic Reviews
imum number of studies necessary for conducting a meta-analysis.6
2012;8:CD007566.
Second, the results of the included meta-analyses may have suffered
11. Montgomery P, Dennis J. Physical exercise for sleep problems in adults
from small-study effects (publication bias, etc.). However, the assess- aged 60+. Cochrane Database of Systematic Reviews 2002;CD003404.
ment of such in the original meta-analyses was not justifiable since
12. Tomlinson D, Diorio C, Beyene J, Sung L. Effect of exercise on cancer-
all three meta-analyses included less than 10 effect sizes and a min- related fatigue: a meta-analysis. American Journal of Physical Medicine &
imum of 10 effect sizes is recommended before any such analyses is Rehabilitation 2014;93:675–686.
performed.44 Third, the current study inherited the potential biases 13. Yang B, Xu J, Xue Q, Wei T, Xu J, Ye C, et al. Non-pharmacological inter-
included in not only the original studies but also the meta-analyses ventions for improving sleep quality in patients on dialysis: systematic
review and meta-analysis. Sleep Medicine Reviews 2014;23C:68–82.
themselves.9,14,15 This includes, but is not necessarily limited to, eco-
logical fallacy, specifically Simpson’s paradox. Finally, while the inclu- 14. Yang PY, Ho KH, Chen HC, Chien MY. Exercise training improves sleep
quality in middle-aged and older adults with sleep problems: a system-
sion of the study by Araghi et al.15 may be questioned given that it was
atic review. Journal of Physiotherapy 2012;58:157–163.
conducted in sleep apnea participants using the more objective AHI,
15. Araghi MH, Chen YF, Jagielski A, Choudhury S, Banerjee D, Hus-
the significance of the change compared to the other included studies sain S, et al. Effectiveness of lifestyle interventions on obstruc-
that used more general and subjective sleep quality measures in partic- tive sleep apnea (OSA): systematic review and meta-analysis. Sleep
ipants that did not appear to have sleep apnea appeared to be similar. 2013;36:1553–1562E.
16. Kubitz KA, Landers DM, Petruzzello SJ, Han M. The effects of acute
and chronic exercise on sleep. A meta-analytic review. Sports Medicine
1996;21:277–291.
5 CONCLUSIONS
17. Iftikhar IH, Kline CE, Youngstedt SD. Effects of exercise training on
sleep apnea: a meta-analysis. Lung 2014;192:175–184.
The results of the current study suggest that exercise is associated
18. Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, Ioan-
with improvements in selected sleep outcomes in the sample of adults nidis JP, et al. The PRISMA statement for reporting systematic
included in the meta-analyses. To increase public health reach, a need reviews and meta-analyses of studies that evaluate health care inter-
exists for a large, well-executed and more inclusive systematic review ventions: explanation and elaboration. Annals of Internal Medicine
2009;151:W65–W94.
with meta-analysis on this topic.
19. Sacks HS, Chalmers TC, Smith H. Randomized versus historical con-
trols for clinical trials. American Journal of Medicine 1982;72:233–240.
CONFLICTS OF INTEREST
20. Schulz KF, Chalmers I, Hayes R, Altman DG. Empirical evidence of
None. bias: dimensions of methodological quality associated with estimates
of treatment effects in controlled trials. JAMA 1995;273:408–412.
21. Burnham JF. Scopus database: a review. Biomedical Digital Libraries
REFERENCES 2006;3:1–8.
1. Institute of Medicine. Sleep disorders and sleep deprivation: an unmet 22. Lee E, Dobbins M, DeCorby K, Mcrae L, Tirilis D, Husson H. An optimal
public health problem. Washington, DC: The National Academies Press; search filter for retrieving systematic reviews and meta-analyses. BMC
2006. Medical Research Methodology 2012;12:51.
2. University of Maryland. Sleep disorders. 1-29-2015. 23. Reference Manager [computer program]. Version 12.0.3. Philadelphia, PA:
3. Glass J, Lanctot KL, Herrmann N, Sproule BA, Busto UE. Sedative hyp- Thompson ResearchSoft; 2009.
notics in older people with insomnia: meta-analysis of risks and bene- 24. Microsoft Excel [computer program]. Version 2007. Redmond, WA:
fits. BMJ 2005;331:1169. Microsoft Corporation; 2010.
4. Smith MT, Perlis ML, Park A, Smith MS, Pennington J, Giles DE, et al. 25. Cohen J. Weighted kappa: nominal scale agreement with provi-
Comparative meta-analysis of pharmacotherapy and behavior therapy sion for scaled disagreement or partial credit. Psychological Bulletin
for persistent insomnia. American Journal of Psychiatry 2002;159:5–11. 1968;70:213–220.
5. Brady TJ, Jernick SL, Hootman JM, Sniezek JE. Public health interven- 26. Cicchetti DV, Sparrow SA. Developing criteria for establishing inter-
tions for arthritis: expanding the toolbox of evidence-based interven- rater reliability of specific items: applications to assessment of adap-
tions. Journal of Women’s Health (Larchmt) 2009;18:1905–1917. tive behavior. American Journal of Mental Deficiency 1981;86:127–137.
6. Higgins JPT, Green S. Cochrane Handbook for Systematic Reviews of 27. Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C,
Interventions Version 5.1.0 [updated March 2011]. 2011. The Cochrane et al. Development of AMSTAR: a measurement tool to assess the
Collaboration. methodological quality of systematic reviews. BMC Medical Research
7. Shorten A, Shorten B. What is meta-analysis?. Evidence-Based Nursing Methodology 2007;7:10.
2013;16:3–4. 28. Shea BJ, Bouter LM, Peterson J, Boers M, Andersson N, Ortiz Z,
8. Smith V, Devane D, Begley CM, Clarke M. Methodology in conducting et al. External validation of a measurement tool to assess systematic
a systematic review of systematic reviews of healthcare interventions. reviews (AMSTAR). PLoS ONE 2007;2:e1350.
BMC Medical Research Methodology 2011;11:15. 29. Shea BJ, Hamel C, Wells GA, Bouter LM, Kristjansson E, Grimshaw J,
9. Chiu HY, Huang HC, Chen PY, Hou WH, Tsai PS. Walking improves et al. AMSTAR is a reliable and valid measurement tool to assess the
sleep in individuals with cancer: a meta-analysis of randomized, con- methodological quality of systematic reviews. Journal of Clinical Epi-
trolled trials. Oncology Nursing Forum 2015;42:E54–E62. demiology 2009;62:1013–1020.
36 GA KELLEY AND KS KELLEY

30. Kang DY, Wu YX, Hu D, Hong Q, Wang JL, Zhang X. Reliability and asymmetry in meta-analyses of randomised controlled trials. BMJ
external validity of AMSTAR in assessing quality of TCM system- 2011;343:d4002.
atic reviews. Evidence-Based Complementary and Alternative Medicine 45. Dersimonian R, Laird N. Meta-analysis in clinical trials. Controlled Clini-
2012;2012:1–7. cal Trials 1986;7:177–188.
31. Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello 46. Comprehensive meta-analysis [computer program]. Version 3.3. New
P, et al. GRADE: an emerging consensus on rating quality of evidence Jersey: Englewood, 2015.
and strength of recommendations. BMJ 2008;336:924–926.
47. Mason M, Welsh EJ, Smith I. Drug therapy for obstructive sleep apnoea
32. Harzing’s Publish or Perish [computer program]. Version 4.27 2007. in adults. Cochrane Database of Systematic Reviews 2013;5:CD003002.
33. Mosteller F, Colditz GA. Understanding research synthesis (meta- 48. Chiu HY, Chiang PC, Miao NF, Lin EY, Tsai PS. The effects of mind-
analysis). Annual Review of Public Health 1996;1–23. body interventions on sleep in cancer patients: a meta-analysis of ran-
34. Borenstein M, Hedges L, Higgins J, Rothstein H. Introduction to meta- domized controlled trials. Journal of Clinical Psychiatry 2014;75:1215–
analysis. West Sussex: John Wiley & Sons; 2009. 1223.
35. Cohen J. A power primer. Psychological Bulletin 1992;112:155–159. 49. Wu WW, Kwong E, Lan XY, Jiang XY. The effect of a meditative
36. Cochran WG. The combination of estimates from different experi- movement intervention on quality of sleep in the elderly: a system-
ments. Biometrics 1954;10:101–129. atic review and meta-analysis. Journal of Alternative & Complementary
Medicine 2015;21:509–519.
37. Higgins JPT, Thompson SG, Deeks JJ, Altman DG. Measuring inconsis-
tency in meta-analyses. BMJ 2003;327:557–560. 50. Schutte-Rodin S, Broch L, Buysse D, Dorsey C, Sateia M. Clinical guide-
line for the evaluation and management of chronic insomnia in adults.
38. Graham PL, Moran JL. Robust meta-analytic conclusions mandate the Journal of Clinical Sleep Medicine 2008;4:487–504.
provision of prediction intervals in meta-analysis summaries. Journal of
Clinical Epidemiology 2012;65:503–510. 51. Physical Activity Guidelines Advisory Committee. Physical Activity Guide-
lines Advisory Report. Washington, DC: U.S. Department of Health and
39. Kelley GA, Kelley KS. Impact of progressive resistance training on lipids Human Services; 2008.
and lipoproteins in adults: another look at a meta-analysis using pre-
diction intervals. Preventive Medicine 2009;49:473–475.
40. Higgins JP, Thompson SG, Spiegelhalter DJ. A re-evaluation of
random-effects meta-analysis. Journal of the Royal Statistical Society:
Series A 2009;172:137–159. SUPPORTING INFORMATION

41. Ades AE, Caldwell DM, Reken S, Welton NJ, Sutton AJ, Dias S. Evidence Additional Supporting Information may be found online in the support-
synthesis for decision making 7: a reviewers checklist. Medical Decision ing information tab for this article.
Making 2013;33:679–691.
42. Cohen J. Statistical power analysis for the behavioral sciences. New York:
Academic Press; 1988. How to cite this article: Kelley GA, Kelley KS. Exer-

43. Egger M, Davey Smith G, Schneider M, Minder C. Bias in meta-analysis

cise and sleep: a systematic review of previous
detected by a simple graphical test. BMJ 1997;315:629–634. meta-analyses. J Evid Based Med. 2017;10:26–36.
44. Sterne JA, Sutton AJ, Ioannidis JP, Terrin N, Jones DR, Lau J, https://doi.org/10.1111/jebm.12236
et al. Recommendations for examining and interpreting funnel plot