European Journal of Epidemiology (2020) 35:49–60

https://doi.org/10.1007/s10654-019-00576-5

GUIDELINES

A 24‑step guide on how to design, conduct, and successfully publish

a systematic review and meta‑analysis in medical research
Taulant Muka1 · Marija Glisic1,2 · Jelena Milic3,4 · Sanne Verhoog1 · Julia Bohlius1 · Wichor Bramer5 ·
Rajiv Chowdhury6 · Oscar H. Franco1

Received: 21 June 2019 / Accepted: 29 October 2019 / Published online: 13 November 2019
© Springer Nature B.V. 2019

Abstract
To inform evidence-based practice in health care, guidelines and policies require accurate identification, collation, and
integration of all available evidence in a comprehensive, meaningful, and time-efficient manner. Approaches to evidence
synthesis such as carefully conducted systematic reviews and meta-analyses are essential tools to summarize specific topics.
Unfortunately, not all systematic reviews are truly systematic, and their quality can vary substantially. Since well-conducted
evidence synthesis typically involves a complex set of steps, we believe formulating a cohesive, step-by-step guide on how
to conduct a systemic review and meta-analysis is essential. While most of the guidelines on systematic reviews focus on
how to report or appraise systematic reviews, they lack guidance on how to synthesize evidence efficiently. To facilitate the
design and development of evidence syntheses, we provide a clear and concise, 24-step guide on how to perform a systematic
review and meta-analysis of observational studies and clinical trials. We describe each step, illustrate it with concrete exam-
ples, and provide relevant references for further guidance. The 24-step guide (1) simplifies the methodology of conducting a
systematic review, (2) provides healthcare professionals and researchers with methodologically sound tools for conducting
systematic reviews and meta-analyses, and (3) it can enhance the quality of existing evidence synthesis efforts. This guide
will help its readers to better understand the complexity of the process, appraise the quality of published systematic reviews,
and better comprehend (and use) evidence from medical literature.

Keywords 24 Steps · Systematic review · Meta-analysis · Guideline · Evidence synthesis

Introduction and meta-analyses has grown exponentially in recent dec-

ades [1], and they have gradually migrated to the top of the
The practice of evidence-based medicine requires up-to- pyramid of what is considered good evidence. Nevertheless,
date syntheses of existing evidence. Systematic reviews systematic reviews and meta-analyses can be large, chal-
and meta-analyses ought to be rigorous and transpar- lenging endeavours that are sensitive to bias and errors. To
ent, and provide empirically derived answers to focused provide accurate answers and limit potential pitfalls they
research questions. The publication of systematic reviews require careful preparation and organisation. Several organ-
ised efforts such as the Cochrane collaboration (founded in
Taulant Muka and Marija Glisic have contributed equally to this
1993) have attempted to regulate and improve the quality
work. and uniformity of systematic reviews [2]. A few guidelines
and textbooks also have been published that offer compre-
Wichor Bramer and Rajiv Chowdhury have contributed equally to hensive descriptions of the methodology [2–4]. However,
this work.
several studies assessing the quality of published systematic
Electronic supplementary material The online version of this reviews have shown that not all systematic reviews are truly
article (https​://doi.org/10.1007/s1065​4-019-00576​-5) contains systematic and that their quality is highly variable [5, 6].
supplementary material, which is available to authorized users. Experienced researchers as well as those who are learn-
ing the methodology can use better guidance and training
* Taulant Muka
[email protected] in how to conduct a systematic review and meta-analysis
Extended author information available on the last page of the article

13
Vol.:(0123456789)
50 T. Muka et al.

[1]. Here we provide a concise, 24-step guide on how to Step‑by‑step guide: the 24 steps
perform a systematic review and meta-analysis.
Step 1: Define research question

Aim and scope To define a research question, first establish in detail the
primary and secondary aims of the study (including poten-
We present a concise and comprehensive practical guide tial effect modifiers). The more clearly a research question
and a checklist with 24 steps that can help biomedical focuses on, and clearly defines the science and summarizes
researchers conduct a systematic review and meta-analy- the aim of the research project, the more it will facilitate
sis (Fig. 1). This guide (1) simplifies the methodology of building the search strategy, whether focused or broad,
a systematic review, (2) provides tools to conduct meth- and conducting the systematic review. Developing a good
odologically sound systematic reviews and meta-analyses, research question and defining the aim of the study requires
and (3) it can enhance the quality of existing evidence scanning the literature to identify gaps in the field. The
synthesis efforts. This guide can be used by anyone plan- existence of systematic reviews on similar research ques-
ning a systematic review, whether one that is solely nar- tions is not an obstacle to another systematic review if new
rative or includes a quantitative element; however, health analysis will close gaps and add value. Some research fields
professionals and researchers who are familiar with basic also develop rapidly; if new publications appear frequently,
methods of research and are able to interpret basic statis- new and more current systematic review of the evidence
tical principles used in health research may benefit most or maintaining living network meta-analyses using automa-
from it. The supplemental material accompanying this tized approaches may be indicated [7]. A valuable research
article provides more detailed information and examples question necessarily emerges from existing knowledge, and
for less experienced researchers. there are tools that may facilitate the definition and analysis
of the research question. These include PICO(S), used in

Fig. 1  24-STEP GUIDE checklist: steps to be followed to successfully design and conduct a systematic review and meta-analysis

13
A 24-step guide on how to design, conduct, and successfully publish a systematic review and… 51

evidence-based clinical practice [8]; PEO [9] and SPICE the full 1000 references (the maximum number of refer-
[10] for qualitative research questions; and SPIDER [11] in ences that is possible to download from Google Scholar)
mixed-methods research. In w1 we discuss in more detail from Google Scholar, but mostly only the first 200 refer-
how to define a research question. ences have to be added from Google Scholar [14]. In case
the work includes synthesis of evidence from clinical trials,
Step 2: Establish the team Cochrane library is recommend to be searched for relevant
references, although our work has shown that all included
A well-organized and coordinated team is necessary. Many references would have been found had we not searched the
steps such as the literature search, revision process, and Cochrane library [14]. PsycINFO and CINAHL databases
quality assessment require double-checking by independent should be searched if the research question is related to the
reviewers, and a third independent reviewer is often needed field of psychiatry, psychology and/or to nursing and allied
to resolve disagreements that may arise during the study health. Research has also shown that in CINAHL, the index-
inclusion process. Choose carefully colleagues and experts ing of qualitative research is better than that in Pubmed,
that you are planning to collaborate with; you should evalu- therefore it is recommended for the search of those study
ate their competence in the field and integrity [12]. The team types as well. Central to search quality and reproducibility
should have members whose expertise spans searching for is the inclusion of a librarian or search specialist [15]. Our
studies (i.e. a librarian or medical information specialist), group has established a method that describes in detail a
understanding primary study methods and systematic review 15-step process to develop a systematic search strategy [16].
methods, synthesizing findings and performing meta-analy- In appendix w2 we offer a few basic recommendations for
sis, and knowing the area under investigation. The complex- searching databases.
ity of the question being addressed and the expected number
of references also will figure in the size of the team. The Example See w3 for an example of the search strategy for
makeup of the team will be established after the final search a recently published meta-analysis of intervention studies,
since the number of hits obtained will determine the partici- which evaluated the association between phytoestrogen sup-
pation of independent reviewers. Expertise should be bal- plementation followed with regular, normocaloric diet and
anced across the team members so that one group of experts glucose homeostasis, and risk of type 2 diabetes in adult
is not overly influential. For example, review teams that are women. A literature search was done using five electronic
too dominated by clinical content experts are more likely to databases: MEDLINE via Ovid, Embase.com, Web of Sci-
hold preconceived opinions related to the topic of the sys- ence Core Collection, Cochrane CENTRAL via Wiley, and
tematic review, spend less time conducting the review, and Google Scholar. This example reflects the complexity of a
produce lower quality reviews [13]. Finally, a team cannot search strategy and presents search syntaxes for the different
function without a team leader. The leader is not by defini- medical databases [16].
tion a professor or the most senior member of the team. The
leader coordinates the project, takes care that study protocol Step 4: Define selection criteria (inclusion/exclusion)
is followed, keeps all team members informed, and facili-
tates their participation in all phases of the project. Selection criteria identify relevant evidence during the
screening process. The selection criteria guide the reviewers,
Step 3: Define the search strategy (steps 3, 4, and 5 are save time, minimize mistakes, and guarantee transparency
done in parallel) and reproducibility. They depend on the research question
and incorporate study characteristics that can include study
A comprehensive search forms the foundation of any sys- design, date of publication, and geographical location; char-
tematic review and consists of writing specific search strate- acteristics of the study population such as age, sex, and pres-
gies in different online databases to retrieve eligible studies. ence of disease; characteristics of the exposure and outcome
Inadequate searches or errors in search strategies may miss measured; and characteristics of the methods used such as
evidence, while untargeted, broad searches lead to superflu- type of analysis, adjustment for confounders, measure of
ous articles and waste time. Missing relevant articles may association reported, etc. An important step in establishing
bias estimates. Numerous online databases can be searched. the selection criteria is the evaluation of the type(s) of study
It is not necessary to search all databases, however, no single design that may best answer the research question. In addi-
database can encompass all medical literature. Recently, our tion to looking for study designs that may yield the highest
group has shown that optimal searches should be performed level of evidence it is important to think about which study
by using Embase, MEDLINE, Web of Science, and Google designs fit the research question. After inclusion/exclusion
Scholar at a minimum to ensure adequate, efficient coverage criteria are established a so-called checklist should be writ-
[14]. In the search it is not necessary however to retrieve ten. A checklist guides the reviewers through the screening

13
52 T. Muka et al.

process and a well written checklist will save time and mini- example of a study protocol in w6. Registering the review is
mize mistakes during screening. A sample checklist can be recommended to avoid overlap and superfluous efforts, and
found in w4. to provide transparency. There are a few platforms for regis-
tration; those most often used are Prospero (http://www.crd.
Example Meta-analyses that include intervention stud- york.ac.uk/PROSPE ​ RO/) for reviews in health or social care,
ies often pool estimates from randomized controlled trials and Cochrane (http://www.cochr​ane.org/cochr​ane-revie​ws)
(RCTs) that use substantially different control groups. For for reviews regarding interventions. Instructions how to reg-
example, a control arm may receive placebo or a control sub- ister a review at Prospero are given in w7.
stance, or the same treatment as the intervention arm but at a
lower dose. In a recent systematic review and meta-analysis Step 7: Run the search strategy in multiple databases
of RCTs we evaluated the association of plant-based thera-
pies with menopausal symptoms. To maintain consistency As mentioned in step 3, a literature search should include
and because of the difficulty of interpreting results without at least four online databases: Embase, MEDLINE, Web of
a placebo or control, we excluded head-to-head trials that Science, and Google Scholar. Each database has its own way
compared nonhormonal therapies with estrogen or other of writing a search strategy.
medications, that lacked a placebo group [17].
Step 8: Collect all references and abstracts in a single file
Step 5: Design data collection form
Collect all of the research results of step 7 in EndNote or
A key step in a systematic review is the extraction of per- another tool such as Covidence (www.covid​ence.org), Dis-
tinent data from primary studies (and not from individual tillerSR (www.disti​llerc​er.com), or Rayyan (rayya​n.qcri.
subjects) using a standardized data extraction form. Data are orgrayyan.qcri.org). If available, we recommend using
collected on (1) general characteristics of the study such as EndNote, which provides support for step 9. To import the
investigator name(s), year of the study, and funding source, selected references from a database into an Endnote file,
(2) characteristics of the study population that may include export the references from that database in a format recog-
age, sex, and ethnicity, (3) exposure or intervention, which nized by EndNote. The instructions on exporting citations
can include assessment method, distribution in the study from the major databases into EndNote are provided in w8.
population, and dosage when describing drugs, (4) out-
comes, (5) methods such as the type of statistical analysis Step 9: Eliminate duplicates
that was used and factors adjusted for, and (6) results such
as measures of association, stratified analyses, and distribu- Retrieving relevant studies from various databases generally
tion of results agreement. Designing the data extraction form leads to articles being identified multiple times. Removal of
requires careful consideration of the research question and duplicate records will reduce the reviewers’ workload when
often benefits from piloting the form in at least five studies screening titles and abstracts. Due to the heterogeneous
in the field before finalization. A variety of software applica- nature of articles in databases, de-duplication can be cum-
tions allow organization of the data extraction form, includ- bersome and time-consuming. Our group has published a
ing Microsoft Access/Excel, Qualtrics, REDCap, Google method using EndNote for faster yet accurate de-duplication
Forms/Sheets, SRDR (Systematic Review Data Repository; [19]. There are other software available for deduplication,
https:​ //srdr.ahrq.gov/home/index)​ etc. [18]. Further, software but they have not been thoroughly evaluated for accuracy.
that could be used for reference screening such as Covidence
(https:​ //www.covide​ nce.org) and DistillerSR (www.distil​ lerc​ Example Detailed instructions on how to perform de-dupli-
er.com) can be also used as data extraction tools and In w5 cation using EndNote can be found elsewhere [19].
an example of a data extraction form is provided.
Step 10: Have at least two reviewers screen title
Step 6: Write the study protocol and register the review and abstract

The study protocol contains the research question, primary The titles and/or abstracts of each reference should be
and secondary aims, study design, inclusion and exclusion screened for relevance by at least two reviewers. It is not
criteria, electronic search strategy, and the analysis plan necessary that any single person screen all references as
described in detail. The study protocol guides the reviewers long as each reference has been screened by two independ-
through the screening process. When writing the protocol, ent reviewers. For example, one person might screen all
relevant experts should be asked to provide feedback and of the references, while for the second screening all of the
make sure the protocol covers all elements. We provide an references may be divided across other reviewers. Titles

13
A 24-step guide on how to design, conduct, and successfully publish a systematic review and… 53

and abstracts can be screened simultaneously, judging the Step 11: Collect, compare, and select for retrieval
relevance of the abstract if the title is found to be relevant.
It is not necessary to screen title first and after that the The references screened by two independent reviewers are
abstract as was done in the past when screening was done then collected and compared. The software tools mentioned
on paper. In this phase, references are selected based on in step 10 all have a comparison feature, and the method
the selection criteria applied to the title and abstract, and described in our previous research describes how this can
not to the full text of the article. If a reference lacks an be performed in EndNote [19]. The overlapping set of ref-
abstract, and has only the title, the reference should be erences that both reviewers have selected to include in the
included for the next step. At this phase, it is not necessary review are considered for the next step: retrieval of the
to keep track of the reason for exclusion [19]. Various soft- full text. For the non-overlapping references on which the
ware applications such as Rayyan, Covidence, and Distill- reviewers did not agree, a meeting between the two review-
erSR are available for the title and abstract screening phase ers should be organized to reach a common final decision.
[20–22]. We do not recommend the use of Excel for this A third independent reviewer should be assigned in cases of
purpose because it is complicated and time-consuming. persisting disagreement (Fig. 2); or, the two reviewers could
Our group has developed a method for screening title and decide to consider each reference included by at least one
abstract using EndNote. The method is very fast with a of the reviewers in the next step. Usually the third reviewer
median of 300 references screened per hour [19]. should be a senior researcher with experience in the topic.
Screening of titles and abstracts can be performed in
other software as well. Rayyan, Covidence distiller sr Step 12: Retrieve full text and apply selection criteria
are tools that offer these services. Rayyan uses artificial
intelligence to determine the highest potential references The full texts of the references selected based on titles and
among those yet to be screened. If this can be trusted, it abstracts are retrieved. Full texts can usually be found using
will reduce the time needed to screen references. How- the “find full text option” in EndNote, via searching local
ever, until now it is not yet trusted, meaning that despite libraries or online search engines such as Google Scholar
the relevance ranking all reference normally will have to and Research Gate, or contacting the authors directly. If
be screened. In our experience most tools require actions the full text of a reference is not directly available to the
for each reference to be included. Our method in Endnote reviewers the reference should not be ignored, but instead
allows multiple references to be excluded at once, which the university library can assist in retrieving a copy of the
greatly reduces the time needed for screening. article via interlibrary loan. Once all full texts are retrieved,
two independent reviewers screen the articles using the

Fig. 2  Reference screening

process

13
54 T. Muka et al.

selection criteria to select those to be included in the sys- the reasons for exclusion citing number of studies excluded
tematic review. A third independent reviewer is available for each reason; and the number of the studies included in
to solve disagreements. As in abstract screening, custom the systematic review. An example of a flowchart can be
groups in Endnote can distinguish various reasons for exclu- found in online w9.
sion, and articles may be assigned to specific groups for
certain sub-questions. Step 16: Apply data collection form (in pairs)

Step 13: Contact experts The next step is to extract the data contained in the included
studies using the predefined collection form of step 5. Using
Contact authors who are experts in the field to identify any the items in the form, two independent reviewers extract
ongoing or missing study, find unpublished but relevant data detailed data from each article. Close attention to the data
(for example, estimates might be provided in an article for extraction process will yield an initial understanding and
a certain outcome but not given for another), or assist recal- description of the shared characteristics of the body of evi-
culation of summary estimates from a published study with dence and will pave the way for the analytic and interpretive
a standardized set of covariates for more meaningful com- process of synthesis to follow. When extracting the data,
bination of the estimates across all studies in meta-analysis. make clear abbreviations, carefully convert all data using
To identify experts, we recommend corresponding with the same unit(s), maintain consistent definitions, and keep
authors of articles selected in step 12. Check whether the content brief.
references suggested by these authors duplicate references
already reviewed and, if they do not, repeat steps 9–13 to Step 17: Evaluate study quality and risk of bias
make a final decision on whether to include the suggested
studies in the review. Evidence and results should be interpreted in light of the
quality of the included studies. The quality of the research
Step 14: Search for additional references encompasses how a study has been conducted (its meth-
odological quality) and how it has been described (report-
In this step, the references assembled at the end of step 12 ing quality and reproducibility). Poor methodological and
and through step 13 should themselves be reviewed for rel- reporting quality of primary studies included in the review
evant studies cited in them (forward search) or by screening may introduce bias and spurious conclusions. Thus, a valid
studies that have cited the articles (backward search). The assessment of study quality by two independent reviewers
abstract and citation database Elsevier Scopus, may facilitate is essential to guarantee accuracy and generalisability. An
this. Detailed instructions can be found at: https:​ //www.elsev​ important aspect of methodological quality is the risk of
ier.com/__data/asset​s/pdf_file/0005/79196​/scopu​s-quick​ bias in the included studies. While its researchers might
-refer​ence-guide​.pdf. have done the best possible study, the study may still be at
The reference lists of previous systematic reviews that are high risk of confounding, selection bias, and information
related to this systematic review should also be searched. bias. There are a number of checklists available to assist in
For references selected from these sources, EndNote may assessing risk of bias in the included studies. The Cochrane
be used in combination with the Scopus or Web of Science Collaboration’s tool RoB 2 [23] and the Newcastle–Ottawa
databases to semi-automatically download the references Scale [24] are the most commonly used to evaluate the risk
into an existing EndNote library [19]. This again requires of bias in RCTs and in prospective observational studies,
another round of checking references and eliminating dupli- respectively. The RoB 2 tool (online Supplement I) evaluates
cates, and retrieving and screening full-texts in keeping with five possible sources of bias [1]: bias arising from the rand-
steps 8 through 12. omization process [2], bias due to deviations from intended
interventions [3], bias due to missing outcome data [4], bias
Step 15: Make the final selection list and draw the flow in measurement of the outcome, and [5] bias in selection
chart of the reported result. The Newcastle–Ottawa Scale (online
Supplement II) uses a star system (with maximum of nine
The articles selected in steps 12, 13, and 14 will become stars) to evaluate three domains: selection of participants,
the final articles included in the review. A well-designed comparability of study groups, and the ascertainment of out-
flowchart will contain information on the number of relevant comes and exposures of interest. Studies that receive a score
citations identified through database searches, experts, and of nine stars are judged to be at low risk of bias, a score of
reference lists; the number of studies excluded based on title seven or eight stars indicates a medium risk, while a score
and abstract search; the number of full texts screened; the of six or less indicates a high risk of bias. Separate tools
number of studies excluded after full text assessment with have been developed to assess risk of bias in nonrandomised

13
A 24-step guide on how to design, conduct, and successfully publish a systematic review and… 55

studies of interventions (ROBINS-I) [25] and in diagnostic the systematic review by summarizing the number of refer-
accuracy studies, such as QUADAS-2 (Quality Assessment ences they found from the search strategy, the number of
of Diagnostic Accuracy Studies 2) [26]. Further, the QUIPS abstracts and full texts they screened, and the final number of
(Quality In Prognosis Studies) tool has been developed to primary studies they included in the review. This process is
assess risk of bias in predictor finding (prognostic factor) summarized in a flowchart flow-chart (w9). Authors should
studies [27]. While recently, the Prediction model study Risk also describe the characteristics such as the populations stud-
Of Bias Assessment Tool (PROBAST) has been developed ied, types of exposures, intervention details, and outcomes
to assess the quality of prediction model studies for develop- of the included studies in a table and in the main text of
ment, validation, or updating of both diagnostic and prog- the manuscript. If meta-analysis is not feasible (see step 20),
nostic models, regardless of the medical domain, type of authors should describe the results of the included studies,
outcome, predictors, or statistical technique used [28]. including the direction and size of effect, effect consistency
The quality of evidence of the included studies in the across studies, and the strength of evidence for the effect.
systematic review/meta-analysis should be clearly reported, Rodgers et al. offer further guidance [30]. An example of a
discussed, and interpreted to provide readers an idea on how table containing study characteristics is presented in w12.
much confidence they can place in the conclusions provided.
An example of how to evaluate the quality of RCTs can be Step 20: To meta‑analyze or not
found in w10.
Prior to this step, systematic reviews and meta-analyses
Step 18: Prepare database for analysis share the same steps. At step 20, the study team must decide
whether the data gathered for each outcome is suitable for
In this step, data from the newly formed database are col- pooling using quantitative methods. By combining data
lated, imported to a statistical evaluation program (SPSS or from different studies, the sample size increases generating
Excel), and prepared for analysis. Analyses may be descrip- more statistical power and improving estimates of the size
tive (step 19), such as a structured summary and discussion of the effect, and it has the potential of resolving uncer-
of the studies’ characteristics, findings, and quality, or they tainty when primary studies disagree. Also, when possi-
may be quantitative (step 20), which involves statistical anal- ble, a meta-analysis makes it easier to describe the pooled
yses (step 21). The data quality depends on the studies that effect of the findings (instead of describing the findings of
are included in the review. In online table w11 we give an each study separately). The decision to pool depends on the
overview of effect measures by type of study and suggested degree of heterogeneity. Heterogeneity is not a condition
random and fixed-effect models which shall be used to pool to ignore, but to report, and it can occur at multiple levels:
the data together. For example, in observational studies prev- study characteristics such as differences in design (interven-
alence, mean difference values, beta regression coefficients, tional or observational); population characteristics including
odds ratios, relative risks, or hazard ratios may be reported. differences in age, gender, and geographical location; and
While clinical trials, when the outcome is dichotomous, may methods and results encompassing differences in analyses,
report baseline and end of the trial values of outcomes, and adjustments, and measures of association. While clinical,
the mean harmonized across included studies. For example, biological or methodological heterogeneity may be specific
some studies may report risk estimates by comparing vari- to certain topic, statistical heterogeneity can be examined
ous extreme quantiles (top vs. bottom fifths, thirds, etc.), per using the same statistical methods across all meta-analyses
unit or per standard deviation change in baseline exposure, [31]. The most commonly used methods to evaluate statisti-
etc. Also, outcomes may be measured on different scales: for cal heterogeneity include the Cochrane’s Chi squared test
blood glucose, mmol/L is the most common measurement (Cochran’s Q), which examines the null hypothesis that all
used in the UK while mg/dL is predominantly used in the studies are evaluating the same effect but may not always
USA and continental Europe. Systematic reviews of health accurately detect heterogeneity. Higgins’s I­ 2 statistic is also
economics outcomes might be presented in different curren- widely used. Higgins’s ­I2 represents the percentage of vari-
cies from different years, and require currency conversions ation between the sample estimates that is due to heteroge-
and adjustments for inflation [29]. There are online tools that neity rather than to sampling error (tells us what proportion
can help with unit and currency conversion. of the total variation across studies is beyond chance) [31].
It can take on values from 0 to 100%, with 100% being the
Step 19: Conduct descriptive synthesis maximum level of heterogeneity. Often ­I2 values below 25%
are considered low, 25 to 50% moderate, and above 75%
Descriptive synthesis relies primarily on words and text to high heterogeneity [32]. ­I2 is routinely implemented in all
summarize and explain findings. Whether including a meta- Cochrane reviews and in meta-analyses published in medical
analysis or not, authors should describe the process flow of journals. However, I­ 2 has some uncertainty, and Higgins and

13
56 T. Muka et al.

Thompson provided methods to calculate this uncertainty, have not been studied in head-to-head studies. Network meta-
while, recently the other investigators indicated that ­I2 has analysis as compared to pairwise meta-analysis, allows the
low statistical power with small numbers of studies and its visualisation of a greater number of evidence, estimation of
95% confidence intervals can be large [33] and given that the relative effectiveness among all interventions, and ranks
­I2 is not precise, 95% confidence intervals should always be ordering of the intervention [39]. The underlying assumption
given [31]. For example, in STATA, it is possible to calculate of NMA is that there are no study or individual’s character-
95% CI using either of two methods: a test based approach istics that would modify the relative treatment effect of each
or a non-central χ2 based approach (heterogi module). The treatment in comparison with other treatments included in the
performance of these two methods is comparable, although meta-analysis [40]. NMA can be performed for continuous
the test based approach often gives lower values for lower and dichotomous RCTs outcomes but also for event rates and
and upper confidence intervals, so that the non-central χ 2 from survival models, using an appropriate scale (mean differ-
based approach may be preferable [34]. The perception of ence, odds ratio, hazard ratio, relative risk). Detailed instruc-
statistical heterogeneity may influence researcher’s decision tions how to perform NMA can be found elsewhere [40].
on whether the data are similar enough to combine different
studies. Therefore, when making a decision on whether or Example We have studied the associations between phytoes-
not to pool treatment estimates in a meta-analysis, Ruecker trogen intake and type 2 diabetes (T2D) risk and glucose
et al., suggest that the between-study variance (τ2), rather homeostasis. We included observational longitudinal stud-
than ­I2 may be appropriate measure for this purpose [35]. ies and RCTs. Although, the estimates reported in obser-
Stratification is a tool to explore sources of heterogeneity vational studies (risk of developing T2D) and RCTs (mean
(see step 21). It is important that the studies in the meta- serum change) could not be pooled together, they are com-
analysis are comparable in terms of definitions, coding, plementary. In particular, the findings of beneficial effect of
methods, comparisons, and categories of exposure between phytoestrogens on T2D risk were supported with findings
studies. Therefore, before synthesizing estimates it is crucial from RCTs where we found that phytoestrogen supplemen-
to use the same estimates and standardize the coding and tation improved glucose homeostasis [16]. Therefore, the
definitions when possible. More details on which potential conclusions of our review were stronger than if would have
problems and how to deal with them while standardising the included only observational studies or solely RCTs. There-
data for meta-analysis can be found in w13. fore, the conclusions of our review were stronger than if
Another doubt a researcher may face is whether or not to would have included only observational studies or solely
synthase different types of studies which address the same RCTs. Similarly, in an another systematic review and meta-
research question (i.e. observational and RCTs or observa- analysis of alcohol intake and onset of menopause, the meta-
tional and experimental studies). The inclusion of more than analysis results of cross-sectional and longitudinal studies
one study design may improve the quality of systematic provided similar conclusions, strengthening the validity of
review significantly and contribute to better understanding, the findings had we choose only cross-sectional studies [41].
easier interpretation of findings and clarification of the contra-
dictory results. Another important uncertainty when studding When synthesising the evidence, authors often need to
health interventions is whether or not to include RCTs only choose between two statistical methods: the fixed effect (FE)
or also non-randomized. Although, RCTs are considered to and the random effects (RE) model. These two methods may
be on the top of quality of evidence pyramid, Ioannidis et al. yield similar or discrepant results. However, even if results
[36] reported that discrepancies between RCT and non-rand- of the two models are similar summary estimates should be
omized studies were less common when only nonrandomized interpreted in a different way [42]. The basic assumption of
studies with a prospective design were considered. Also, the the FE model is that the exposure or treatment effect under
Cochrane Collaboration offers a guide for inclusion of nonran- observation is fixed in all studies included in the meta-analy-
domized studies [37] and has developed a tool for assessing sis, whereas the RE model allows the exposure effect to vary
the risk of bias in both RCT and controlled nonrandomized across the studies. In simple terms, RE model allows the true
studies [38]. Therefore, it is of high interest not to neglect non- effects underlying the studies to differ and thus accounts for
randomized studies especially in cases where randomization unexplained heterogeneity between studies. The main miscon-
may pose important ethical issues. Further, in clinical practice ception is that the model should be chosen based on the test
there are more than two interventions of interest for a single of heterogeneity. Indeed, often when heterogeneity variance is
health condition and researchers often aim to determine the estimated to be 0% the results are identical under the two meta-
best available intervention in a single, coherent analysis of all analysis models. However, the choice of the model should not
the relevant RCTs [7, 39, 40]. A pairwise meta-analysis and its be made based on the test of heterogeneity since heterogeneity
extension network meta-analysis (NMA) have been developed may exist even if it remains undetected by the test. In Fig. 3 we
to facilitate indirect comparisons of multiple interventions that

13
A 24-step guide on how to design, conduct, and successfully publish a systematic review and… 57

compare the FE and RE models and give instructions on how Step 21: Exploration of heterogeneity
to choose a model suitable for your analysis [42].
Finally, various software applications are able to perform Subgroup analyses, or stratification, should be taken into con-
a meta-analysis. However, one of the most commonly used sideration from step 1 within the definition of primary and
and for the inexperienced researcher perhaps one of the sim- secondary aims. Factors by which results might differ—that
plest for meta-analysis is metan command in STATA [43]. is, effect modifiers—often include study characteristics such
Guidance on how to undertake meta-analysis using Stata is as study design, geographical location, date of publication, and
provided by Chaimani et al. [44] In case authors do not have type of intervention, and also population characteristics such
a STATA subscription, meta-analysis packages are available as age, gender, ethnicity, and presence of disease [45]. Results
in the open access statistical environment R (Metafor (R should be presented and pooled by different categories of these
package)) [31]. For users who are not experiences with using factors to compare whether the pooled estimates differ within
R, we suggest JASP or Jamovi which are free, open-source groups, and whether t­au2 changes. Heterogeneity should also
programs used to perform statistical analysis tests by using R be evaluated within specific strata. Meta-regression analysis
packages. Further, Review Manager (RevMan) developed by can also be used to explore whether observed heterogeneity
the Cochrane Collaboration may be a good choice for those is a consequence of the specific study or population charac-
who are new to the world of meta-analysis. Nevertheless, in teristics. Meta-regression is therefore similar to conventional
order to perform a simple meta-analysis it is possible to use statistical regression used to determine the effect of one factor
Excel add-on such as MetaEasy or MetaXL [34]. upon an outcome variable. Meta-regression is often done when

Fig. 3  Fixed versus random effects model

13
58 T. Muka et al.

more than 10 studies are included in a meta-analysis [46]. An bases judgment of the quality of evidence on the magnitude
example of subgroup analyses is in w14. Recommendations for of effect, and consideration of the risk of bias, the study
the interpretation of subgroup analyses in systematic reviews design, and consistency and directness of the findings. It
can be found elsewhere [47]. grades evidence as high, moderate, low, or very low. RCTs
start as high quality and observational studies start as low
Step 22: Check reporting bias quality. Limitations in study quality, important inconsistency
of results, or uncertainty about the directness of the evidence
Publication bias occurs whenever the published literature can lower the grade of evidence. Also, certain factors such
is systematically unrepresentative of all completed studies as evidence of a dose–response gradient or strong evidence
[48]. Publication bias originates in a decision to publish that of association based on consistent evidence from two or
is influenced by an experimental or research study’s out- more observational studies with no plausible confounders
come. Most commonly, negative results or those judged not may increase the grade [54]. The evaluation should be per-
significant are less likely to be submitted and accepted for formed independently by two reviewers, while any disagree-
publication. Publication bias is usually evaluated through ment should be discussed with a third, independent reviewer.
a funnel plot in which asymmetry may be assessed visu- Detailed instructions how to use the GRADE approach are
ally, and by using the Egger test. A funnel plot is a scatter given in the online tutorial found at:
plot of the exposure effect estimates from individual studies https​://gdt.grade​pro.org/app/handb​ook/handb​ook.html.
against a measure of study precision (typically the stand-
ard error) [49]. If the funnel is asymmetric, it can imply Step 24: Update, report, and submit for publication
that there are studies missing from the literature. However,
publication bias is not the only cause of funnel plot asym- When ready to submit the study for publication, if the inter-
metry; other causes of bias include heterogeneity, selective val since beginning the search of bibliographic databases is
outcome reporting, and simply chance [49]. Particularly greater than 6–12 months the search should be updated to
when representing a low number of studies a funnel plot identify recently published articles.
may not detect publication bias [50, 51]. Harbord developed Guidelines exist on how to report a systematic review
a modified version of the Egger test for small-study effects and meta-analysis facilitating transparency, reproducibility,
in meta-analysis of controlled trials with binary endpoints and comparability between studies. PRISMA, QUOROM
[52]. Yet, this test is not recommended in meta-analyses of (which evolved into PRISMA), and MOOSE are flowcharts
cohort studies where there is large imbalance in the group that graphically describe the sequence of reporting a system-
sizes; however, in this situation the original Egger test will atic review and meta-analysis. When submitting the study,
often perform well. Further, Begg proposed a bias indica- it is essential to add as an attachment a detailed PRISMA
tor using Kendall’s method (testing the interdependence of or MOOSE report. PRISMA and MOOSE flowcharts are
variance and effect size). This bias indicator makes fewer provided in online Supplements III and IV.
assumptions than that of Egger and in case of small number Finally, additional experts with content expertise may be
of studies, bias cannot be ruled out if the test is not sig- invited to review and comment on the manuscript (and the
nificant. Yet, this test may be used as an exploratory tool published work should acknowledge their assistance). It is
for meta-analysis, as a formal procedure to complement the still possible to improve the quality of the publication further
funnel-plot graph [53]. When the degree of between-trial by appraising the interpretation of the results one last time.
heterogeneity is large, none of the three mentioned tests has
uniformly good properties [52]. Finally, the presence of pub-
lication bias requires reporting and thorough discussion, but Concluding remarks
it need not prevent publishing the study. More information
can be found in w15. Evidence syntheses constitute essential tools for evidence-
based medicine and policy-making in a time of proliferating
Step 23: Check the quality of the evidence: the confidence scientific publications and journals. Healthcare professionals
in the results presented and researchers must understand the principles of preparing
such reviews and follow strict protocols to use them effec-
The strength of the results reported in a systematic review tively. This 24-step guide can simplify the process of con-
and meta-analysis relies, first, upon the quality of the ducting a systematic review, provide healthcare professionals
review’s evidence. Authors can apply the Grading of Rec- and researchers with the tools to conduct methodologically
ommendations Assessment, Development and Evalua- sound systematic reviews and meta-analyses, and enhance
tion (GRADE) approach to score the quality of evidence the quality of synthesis efforts already underway. The guide
included in the systematic review. The GRADE approach will increase readers’ understanding of the complexity of

13
A 24-step guide on how to design, conduct, and successfully publish a systematic review and… 59

the process and the quality of published systematic reviews, 14. Bramer WM, Rethlefsen ML, Kleijnen J, Franco OH. Optimal
and enhance the incorporation of knowledge synthesis into database combinations for literature searches in systematic
reviews: a prospective exploratory study. Syst Rev. 2017;6(1):245.
clinical decisions and policy-making. https​://doi.org/10.1186/s1364​3-017-0644-y.
15. Rethlefsen ML, Farrell AM, Osterhaus Trzasko LC, Brigham
Acknowledgements We would like to thank Georgia Salanti for the TJ. Librarian co-authors correlated with higher quality reported
critical revision of the manuscript, Christopher Owen Ritter for Eng- search strategies in general internal medicine systematic reviews.
lish language editing, and 24-desig​n.com for help with figures’ design. J Clin Epidemiol. 2015;68(6):617–26. https​://doi.org/10.1016/j.
jclin​epi.2014.11.025.
Compliance with ethical standards 16. Glisic M, Kastrati N, Gonzalez-Jaramillo V, et al. Associations
between phytoestrogens, glucose homeostasis, and risk of diabe-
tes in women: a systematic review and meta-analysis. Adv Nutr.
Conflict of interest The authors declare no conflict of interest related 2018;9(6):726–40. https​://doi.org/10.1093/advan​ces/nmy04​8.
to this work. 17. Franco OH, Chowdhury R, Troup J, et al. Use of plant-based ther-
apies and menopausal symptoms: a systematic review and meta-
analysis. JAMA. 2016;315(23):2554–63. https​://doi.org/10.1001/
jama.2016.8012.
References 18. Elamin MB, Flynn DN, Bassler D, et al. Choice of data extrac-
tion tools for systematic reviews depends on resources and review
1. Manchikanti L. Evidence-based medicine, systematic reviews, and complexity. J Clin Epidemiol. 2009;62(5):506–10. https​://doi.
guidelines in interventional pain management, part I: introduction org/10.1016/j.jclin​epi.2008.10.016.
and general considerations. Pain Physician. 2008;11(2):161–86. 19. Bramer WM, Milic J, Mast F. Reviewing retrieved references for
2. Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF. inclusion in systematic reviews using EndNote. J Med Libr Assoc.
Improving the quality of reports of meta-analyses of randomised 2017;105(1):84–7. https​://doi.org/10.5195/jmla.2017.111.
controlled trials: the QUOROM statement. Qual Rep Meta-Anal 20. Mourad Ouzzani HH, Fedorowicz Zbys, Elmagarmid Ahmed.
Lancet. 1999;354(9193):1896–900. Rayyan—a web and mobile app for systematic reviews. Syst Rev.
3. Moher D, Liberati A, Tetzlaff J, Altman DG, Group P. Preferred 2016;5:210. https​://doi.org/10.1186/s1364​3-016-0384-4.
reporting items for systematic reviews and meta-analyses: the 21. DistillerSR EP, Ottawa, Canada, Available at https​://www.evide​
PRISMA statement. Ann Intern Med. 2009;151(4):264–9. ncepa​rtner​s.com/.
4. Dekkers OM, Vandenbroucke JP, Cevallos M, Renehan AG, Alt- 22. Covidence. Cochrane Community. https​://commu​nity.cochr​ane.
man DG, Egger M. COSMOS-E: guidance on conducting system- org/help/tools​-and-softw​are/covid​ence. Accessed 3 Jul 2018.
atic reviews and meta-analyses of observational studies of etiol- 23. Higgins JPT, Savovic J, Page MJ, Hróbjartsson A, Boutron I,
ogy. PLoS Med. 2019;16(2):e1002742. https​://doi.org/10.1371/ et al. A revised tool for assessing risk of bias in randomized tri-
journ​al.pmed.10027​42. als. Cochrane Database Syst Rev. 2016;10(Suppl 1):29–31.
5. Conway A, Inglis SC, Chang AM, Horton-Breshears M, Cle- 24. Higgins JP, Altman DG, Gotzsche PC, et al. The cochrane col-
land JG, Clark RA. Not all systematic reviews are systematic: laboration’s tool for assessing risk of bias in randomised trials.
a meta-review of the quality of systematic reviews for non- BMJ (Clin Res Ed.). 2011;343:d5928. https​://doi.org/10.1136/
invasive remote monitoring in heart failure. J Telemed Telecare. bmj.d5928​.
2013;19(6):326–37. https:​ //doi.org/10.1177/135763​ 3X135​ 03427​ . 25. Sterne JA, Hernan MA, Reeves BC, et al. ROBINS-I: a tool for
6. Gurevitch J, Koricheva J, Nakagawa S, Stewart G. Meta- assessing risk of bias in non-randomised studies of interventions.
analysis and the science of research synthesis. Nature. BMJ (Clin Res Ed). 2016. https​://doi.org/10.1136/bmj.i4919​.
2018;555(7695):175–82. https​://doi.org/10.1038/natur​e2575​3. 26. Whiting PF, Rutjes AW, Westwood ME, et al. QUADAS-2: a
7. Nikolakopoulou A, Mavridis D, Furukawa TA, et al. Living net- revised tool for the quality assessment of diagnostic accuracy
work meta-analysis compared with pairwise meta-analysis in com- studies. Ann Intern Med. 2011;155(8):529–36. https ​ : //doi.
parative effectiveness research: empirical study. BMJ (Clin Res). org/10.7326/0003-4819-155-8-20111​0180-00009​.
2018;360:k585. https​://doi.org/10.1136/bmj.k585. 27. Hayden JA, van der Windt DA, Cartwright JL, Cote P, Bombar-
8. Schardt C, Adams MB, Owens T, Keitz S, Fontelo P. Utilization dier C. Assessing bias in studies of prognostic factors. Ann Intern
of the PICO framework to improve searching PubMed for clinical Med. 2013;158(4):280–6. https:​ //doi.org/10.7326/0003-4819-158-
questions. BMC Med Inform Decis Mak. 2007;7:16. https​://doi. 4-20130​2190-00009​.
org/10.1186/1472-6947-7-16. 28. Wolff RF, Moons KGM, Riley RD, et al. PROBAST: a tool to
9. Bettany-Saltikov J. How to do a systematic literature review in assess the risk of bias and applicability of prediction model stud-
nursing: a step-by-step guide. Berkshire: McGraw-Hill Education; ies. Ann Intern Med. 2019;170(1):51–8. https​://doi.org/10.7326/
2012. M18-1376.
10. Booth Andrew. Clear and present questions: formulating questions 29. Luhnen M, Prediger B, Neugebauer EAM, Mathes T. System-
for evidence based practice. Library Hi Tech. 2006;24(3):355–68. atic reviews of health economic evaluations: a protocol for a sys-
11. Cooke A, Smith D, Booth A. Beyond PICO: the SPIDER tematic review of characteristics and methods applied. Syst Rev.
tool for qualitative evidence synthesis. Qual Health Res. 2017;6(1):238. https​://doi.org/10.1186/s1364​3-017-0639-8.
2012;22(10):1435–43. https:​ //doi.org/10.1177/104973​ 23124​ 5293​ 30. Rodgers M, Sowden A, Petticrew M, et al. Testing methodologi-
8. cal guidance on the conduct of narrative synthesis in systematic
12. Haynes AS, Derrick GE, Redman S, et al. Identifying trustwor- reviews: effectiveness of interventions to promote smoke alarm
thy experts: how do policymakers find and assess public health ownership and function. Evaluation. 2009;15(1):49–73. https​://
researchers worth consulting or collaborating with? PLoS ONE. doi.org/10.1177/13563​89008​09787​1.
2012;7(3):e32665. https:​ //doi.org/10.1371/journa​ l.pone.003266​ 5. 31. Ioannidis JP, Patsopoulos NA, Evangelou E. Uncertainty in
13. Oxman AD, Guyatt GH. The science of reviewing research. Ann heterogeneity estimates in meta-analyses. BMJ (Clin Res ed.).
NY Acad Sci. 1993;703:125–33 discussion 33–4. 2007;335(7626):914–6. https:​ //doi.org/10.1136/bmj.39343.​ 40844​
9.80.

13
60 T. Muka et al.

32. Higgins JP, Thompson SG, Deeks JJ, Altman DG. Meas- 43. Harris RBM, Deeks J, et al. Metan: fixed-and random-effects
uring inconsistency in meta-analyses. BMJ (Clin Res meta-analysis. Stata J. 2008. https:​ //doi.org/10.1177/153686​ 7X08​
Ed.). 2003;327(7414):557–60. https ​ : //doi.org/10.1136/ 00800​102.
bmj.327.7414.557. 44. Chaimani A, Mavridis D, Salanti G. A hands-on practical tuto-
33. Huedo-Medina TB, Sanchez-Meca J, Marin-Martinez F, Botella rial on performing meta-analysis with Stata. Evid-Based Mental
J. Assessing heterogeneity in meta-analysis: Q statistic or I2 Health. 2014;17(4):111–6. https:​ //doi.org/10.1136/eb-2014-10196​
index? Psychol Methods. 2006;11(2):193–206. https ​ : //doi. 7.
org/10.1037/1082-989X.11.2.193. 45. Higgins JPT GSe. Cochrane Handbook for Systematic Reviews of
34. Higgins JP, Thompson SG. Quantifying heterogeneity in a meta- Interventions Version 5.1.0 [updated March 2011]. The Cochrane
analysis. Stat Med. 2002;21(11):1539–58. https:​ //doi.org/10.1002/ Collaboration, 2011. Available from www.handb​ook.cochr​ane.
sim.1186. org.
35. Rucker G, Schwarzer G, Carpenter JR, Schumacher M. 46. Thompson SG, Higgins JP. How should meta-regression analyses
Undue reliance on I(2) in assessing heterogeneity may mis- be undertaken and interpreted? Stat Med. 2002;21(11):1559–73.
lead. BMC Med Res Methodol. 2008;8:79. https ​ : //doi. https​://doi.org/10.1002/sim.1187.
org/10.1186/1471-2288-8-79. 47. Marty Richardsona P, Doneganb S. Interpretation of subgroup
36. Ioannidis JP, Haidich AB, Pappa M, et al. Comparison of evi- analyses in systematic reviews: a tutorial. Clin Epidemiol Glob
dence of treatment effects in randomized and nonrandomized Health. 2019;7:192–8.
studies. JAMA. 2001;286(7):821–30. https​://doi.org/10.1001/ 48. Hannah R, Rothstein AJS, Borenstein M. Publication bias in meta-
jama.286.7.821. analysis. New York: Wiley; 2005.
37. Reeves BCDJ, Higgins JPT. Wells GA Chapter 13: Including non- 49. Mavridis D, Salanti G. Exploring and accounting for publication
randomized studies. In: Higgins JPT, Green S, editors. Cochrane bias in mental health: a brief overview of methods. Evid Based
Handbook for Systematic Reviews of Interventions Version 510 Ment Health. 2014;17(1):11–5. https​://doi.org/10.1136/eb-2013-
[updated March 2011]. Oxford: The Cochrane Collaboration; 10170​0.
2011. 50. Lau J, Ioannidis JP, Terrin N, Schmid CH, Olkin I. The case of the
38. Abraham NS, Byrne CJ, Young JM, Solomon MJ. Meta-analysis misleading funnel plot. BMJ (Clin Res Ed). 2006;333(7568):597–
of well-designed nonrandomized comparative studies of surgi- 600. https​://doi.org/10.1136/bmj.333.7568.597.
cal procedures is as good as randomized controlled trials. J Clin 51. Sutton AJ, Higgins JP. Recent developments in meta-analysis. Stat
Epidemiol. 2010;63(3):238–45. https​://doi.org/10.1016/j.jclin​ Med. 2008;27(5):625–50. https​://doi.org/10.1002/sim.2934.
epi.2009.04.005. 52. Harbord RM, Egger M, Sterne JA. A modified test for small-study
39. Tonin FS, Rotta I, Mendes AM, Pontarolo R. Network meta- effects in meta-analyses of controlled trials with binary end-
analysis: a technique to gather evidence from direct and indirect points. Stat Med. 2006;25(20):3443–57. https​://doi.org/10.1002/
comparisons. Pharm Pract (Granada). 2017;15(1):943. https:​ //doi. sim.2380.
org/10.18549​/Pharm​Pract​.2017.01.943. 53. Begg CB, Mazumdar M. Operating characteristics of a rank corre-
40. Dias S, Caldwell DM. Network meta-analysis explained. Arch lation test for publication bias. Biometrics. 1994;50(4):1088–101.
Dis Child Fetal Neonatal Ed. 2019;104(1):F8–12. https​://doi. 54. Schünemann H, Brożek J, Guyatt G, Oxman A, editors. GRADE
org/10.1136/archd​ischi​ld-2018-31522​4. handbook for grading quality of evidence and strength of rec-
41. Taneri PE, Kiefte-de Jong JC, Bramer WM, Daan NM, Franco ommendations. Updated October 2013. The GRADE Working
OH, Muka T. Association of alcohol consumption with the onset Group, 2013. Available from guidelinedevelopment.org/hand-
of natural menopause: a systematic review and meta-analysis. book. 2013.
Hum Reprod Update. 2016;22(4):516–28. https:​ //doi.org/10.1093/
humup​d/dmw01​3. Publisher’s Note Springer Nature remains neutral with regard to
42. Nikolakopoulou A, Mavridis D, Salanti G. Demystifying fixed jurisdictional claims in published maps and institutional affiliations.
and random effects meta-analysis. Evid-Based Mental Health.
2014;17(2):53–7. https​://doi.org/10.1136/eb-2014-10179​5.

Affiliations

Taulant Muka1 · Marija Glisic1,2 · Jelena Milic3,4 · Sanne Verhoog1 · Julia Bohlius1 · Wichor Bramer5 ·
Rajiv Chowdhury6 · Oscar H. Franco1

1 5
Institute of Social and Preventive Medicine (ISPM), Medical Library, Erasmus MC, Rotterdam, The Netherlands
University of Bern, Mittelstrasse 43, 3012 Bern, Switzerland 6
Department of Public Health and Primary Care, School
2
Swiss Paraplegic Research, Nottwil, Switzerland of Clinical Medicine, University of Cambridge, Cambridge,
3 UK
Department of Medical Informatics and Biostatistics,
Institute of Public Health of Serbia, Belgrade, Serbia
4
Department of Epidemiology, Erasmus MC, Rotterdam,
The Netherlands

13