Literature review designs

Narrative Review
Systematic Review
Meta-analysis
1

2006

Literature review
Defined as a systematic, explicit, and
reproducible way of identifying, evaluating,
and interpreting all of the research findings
and scholarly work available on a topic
A high-quality review is not haphazard
Ideally, all of the existing work should be
included

Considered descriptive or observational

Evidence-based

2006-7

The three types of

literature review designs
Narrative Review

Selective review of the literature that broadly covers

a specific topic.
Does not follow strict systematic methods to locate
and synthesize articles.

Systematic Review

Utilizes exacting search strategies to make certain

that the maximum extent of relevant research has
been considered.
Original articles are methodologically appraised and
synthesized.

Meta-analysis

Quantitatively combines the results of studies that

are the result of a systematic literature review.
Capable of performing a statistical analysis of the
pooled results of relevant studies.

Evidence-based

2006-7

Narrative reviews
Summarize in general what is in the
literature on a given topic
Often written by experts in a given field
A good source for background information

Do not follow strict systematic methods like

the other literature review designs
Therefore, they are prone to bias
Lower in the hierarchy of evidence
Evidence-based

2006-7

Narrative reviews (cont.)

Authors like to write them because they
are relatively easy to carry out and
compose
Practitioners like to read them because
they are easier to comprehend than more
complex designs
Do not require a lot of background knowledge
to understand the message
Evidence-based

2006-7

Narrative reviews
are prone to bias
They do not employ many of the
safeguards needed to control against bias
Authors may be selective as to which articles
are included
They may include articles that support their
hypothesis and exclude those that do not
Rigorous appraisal methods are not used to
evaluate included articles
Evidence-based

2006-7

Prone to bias (cont.)

During the literature search
Authors have their own opinions on the topic
and may try to find studies that support their
viewpoint and overlook conflicting studies

During the synthesis of the literature

The approach to analyzing the collected
information is often subjective and
disorganized
Evidence-based

2006-7

Prone to bias (cont.)

In the discussion and conclusion
The authors opinions may be mixed together
with evidence
Authors may simply count the number of
studies on each side of an issue and then
espouse the view presented by the majority
without considering the strength of each study
e.g., quality, research design used, the effect size,
and sample size
Evidence-based

2006-7

Selection bias in
narrative reviews
a.k.a., reference bias
Occurs when authors choose articles that
support their own conclusions and exclude
articles with conflicting views
Results in an erroneous representation of the
literature
The review may lead undiscerning readers
astray
Evidence-based

2006-7

Systematic reviews
Use strict methods to locate, appraise and
synthesize all research on a topic
Similar to narrative reviews, but with improved
procedural quality

Designed to answer specific clinical

questions where several primary studies
exist
Thus, a good source of clinical evidence
Evidence-based

10

2006-7

Systematic reviews (cont.)

Articles are evaluated using appraisal
instruments
In an attempt to achieve impartiality
More reproducible

Dissimilarities between the findings of

studies are investigated
Multiple reviewers are usually involved
Any disagreements are resolved
Evidence-based

11

2006-7

Systematic reviews (cont.)

The search and selection criteria for

articles are well defined
Multiple databases should be searched
Explicit inclusion criteria

The results of the included studies are

qualitatively or quantitatively synthesized
Qualitative written information is merged
Quantitative data are merged
Evidence-based

12

2006-7

Systematic reviews (cont.)

Because of strict methodology and

thoroughness, conclusions are typically
less biased than narrative reviews
Nevertheless, they may still be influenced
by the authors opinions
Still a potential for selection bias
Criteria may be applied differently when
appraising included studies
Therefore, must be critically appraised
Evidence-based

13

2006-7

Systematic reviews (cont.)

No widely accepted method exists for

assessing the validity of studies
i.e., the process is a judgment call
Thus, reviewers sometimes disagree

Information derived from a systematic

review may be too narrow to completely
answer a specific clinical question
Given that they are designed to answer
focused questions about patient care
Evidence-based

14

2006-7

Differences between narrative and systematic reviews

Feature

Narrative Review
Topic Typically broad-scoped

Data sources and The search strategy and

search strategy databases that were used may
not be provided
Authorship A recognized expert(s) on the
topic
Article selection Typically not specified
criteria
Searching May be extensive, intended to
locate literature on the topic
area in question
Appraisal of Indefinite, may be variable
included articles
Synthesis A qualitative summary is
usually provided
Inferences Sometimes evidence-based
Evidence-based
15

Systematic Review
Focused research question
The search strategy is explicit and
comprehensive with a list of all
databases that were utilized
A team of experts having methodologic
and clinical expertise
Consistently applied inclusion and
exclusion criteria
Extensive, intended to locate all
primary studies on a particular research
question
Critical appraisal is meticulous,
typically involving the use of
data extraction forms
A qualitative summary is provided,
quantitative when the data can be
pooled
Usually evidence-based

2006-7

Systematic reviews
typically only include RCTs
Studies are included primarily based on
their quality
Most systematic reviews only include RCTs
because it is the only design that adequately
controls for confounding variables and biases
The potential for studies to overrate the
treatment effect is higher when bias is present

Studies using other less rigorous designs

are usually eliminated
Evidence-based

16

2006-7

Conducting systematic reviews

Evidence-based

17

2006-7

Search strategy
Should be described in enough detail so
that another researcher could replicate the
results, including:
Database(s) searched
Date the search was performed
Time-frame encompassed by the search
A list of search terms used
Languages
Evidence-based

18

2006-7

Search strategy (cont.)

Conference proceedings, unpublished
studies, and hand-searching of journals
are sometimes included in the search
If so, the procedures involved should be
described
Authors must justify using unpublished work

Evidence-based

19

2006-7

Weighting of studies
Articles may be rejected in a systematic
review due to their poor quality
Alternatively, studies are
assigned weights in relation
to their assessed validity
Studies that are more valid will have more
influence on the reviews final results
Based on methodological quality, width of the
confidence intervals, and external validity
Evidence-based

20

2006-7

Publication bias
Studies with statistically significant results
are more likely to get published than those
with non-significant results
Causes of publication bias:
The author or funding source does not
consider a failed study worthy of submission
Journals are less likely to publish studies that
fail to show positive results
Evidence-based

21

2006-7

Publication bias (cont.)

Reviews affected by this bias tend to give
an overoptimistic view of the effectiveness
of the therapy
The chance of this bias occurring is reduced
when authors of systematic reviews search
sources other than journals

Publication bias in situ

A type of bias where a portion of a studys
results are suppressed
Evidence-based

22

2006-7

Meta-analysis
a.k.a., quantitative systematic review
A type of systematic review that
statistically combines the results from a
number of studies
Capable of producing a single estimate of
the effect of a treatment
Represents the average treatment effect
An estimate of the true treatment effect size
Evidence-based

23

2006-7

Meta-analysis (cont.)
The same explicit methods as systematic
reviews are utilized
Systematic reviews and meta-analyses
are are at the top of the hierarchy of
evidence because of their strict
methodology

Evidence-based

24

2006-7

Weighted average
Meta-analyses typically produce a
weighted average for the treatment effect
estimate
Small samples are more susceptible to
chance variations than larger studies
Thus, they are given less weight than larger
studies so they will have less influence on the
final estimate
Evidence-based

25

2006-7

Weighting (cont.)
Weighting is also based on study quality
The quality of the individual studies is rated
and resulting numeric scores are calculated
A corresponding weight is assigned for each
study prior to analysis

Evidence-based

26

2006-7

Meta-analyses can
increase power
Data from individual studies are combined,
which in effect increases sample size
Chiropractic studies commonly involve too
few subjects to detect true differences
between the groups
Pooling data reduces the potential for type II
error
More likely to detect a treatment effect, if
there actually is one
Evidence-based

27

2006-7

Homogeneity and
heterogeneity
Homogeneity
Similarities of included studies that allow them
to be compared
Homogeneity is preferred in meta-analyses
Achieved by using suitable inclusion criteria

Heterogeneity
Dissimilarities of studies that hamper or even
prevent a realistic comparison of studies
Evidence-based

28

2006-7

Factors that contribute

to heterogeneity
Heterogeneity in the study samples
Caused by conflicting inclusion and exclusion
criteria, differences in patients baseline health
status, dissimilar geographical locations of
groups, etc.

Heterogeneity in the study design

E.g., the way dropouts were managed in the
statistical analysis or the length of time
allowed for patient follow-up
Evidence-based

29

2006-7

Factors that contribute

to heterogeneity (cont.)
The way patients were handled
Regarding comorbid conditions, handling of
complications, the control practitioners had in
patient care, or the outcome measures used

Statistical heterogeneity
When the observed treatment effects of
studies are more dissimilar than what would
be expected by chance
Evidence-based

30

2006-7

Consequences of
heterogeneity
When the results of studies in a metaanalysis are inconsistent, it reduces
confidence in its conclusions
The meta-analysis may actually be
worthless if too dissimilar
For instance, combining studies that used
different types of comparison groups
Or outcomes that were dissimilar
Evidence-based

31

2006-7

Forest plot
A type of graph often used in metaanalyses to illustrate the treatment effect
sizes of the studies
Each study is represented by a black
square that is an estimate of their effect
sizes
A horizontal line extends to either side of
the squares, the 95% confidence interval
Evidence-based

32

2006-7

Forest plot

Evidence-based

33

2006-7

Interpreting a forest plot

If a studys 95% CI crosses over the
vertical line, it is not statistically significant
A diamond with a CI line is sometimes
presented at the bottom of the forest plot
to represent an overall estimate
The black squares may vary in size
representing the weights of the studies
Evidence-based

34

2006-7

Weighting and overall effect

Evidence-based

35

2006-7

Effect size
The difference between the means of the
treatment and control groups
When studies are combined in a metaanalysis, the units of measurement are not
always comparable
Effect sizes are standardized to resolve
this problem producing the standardized
mean difference
Evidence-based

36

2006-7

Standardized mean difference

The effect size divided by the pooled
standard deviation
Pooled standard deviation has been adjusted
for the differences in the sizes of the groups

Represents the standardized difference

between group means
i.e., the relative magnitude of the
experimental treatment
Evidence-based

37

2006-7

Cohens d

Evidence-based

38

2006-7

Odds ratio (OR)

Cohens d is appropriate with continuous
data
An OR is appropriate when the studys
outcome measure is dichotomous
e.g., pain versus no pain

OR is a comparison of the odds of the

outcome being present in the treatment
group against the control group
Evidence-based

39

2006-7

Relative risk (RR)

a.k.a., risk ratio
A comparison of the risk of having the
outcome in the treatment group with that
of the control group
Sometimes OR is reported and sometimes
RR
Experts do not agree on which is most
appropriate
Evidence-based

40

2006-7

OR vs. RR
Consider a hypothetical lower back pain
study with 25 patients in each group
5 in treatment group and 10 in control
group are still in pain at the studys end
OR =
RR =

5/20
10/15
5/25
10/25

Evidence-based

=
=

.25
.66
.2
.4

= .38
= .5
41

2006-7

Meta-analyses are most

valid with RCTs
However, about half of meta-analyses
include observational studies
Primarily cohort and case-control

Observational studies are much more

susceptible to biases and confounding
than RCTs
Therefore, it is usually inappropriate to
statistically combine the results of such
studies
Evidence-based

42

2006-7

Subgroup analysis
Meta-analyses typically include patients
with a variety of characteristics
e.g., age, gender, condition severity, patient
history, etc.

Patients in these subgroups may respond

to treatment differently
e.g., low back pain patients with leg pain may
respond to treatment differently than low back
pain only patients
Evidence-based

43

2006-7

Subgroup analysis (cont.)

Carried out to identify variation between
patient groups regarding certain outcomes
or findings
The process helps readers to distinguish
the effects of a treatment between
subgroups
The statistical power of the subgroups will
decline as a result
Evidence-based

44

2006-7

Meta-regression
A statistical procedure that adjusts for
differences between studies in metaanalyses
May be used in subgroup analyses

Similar to simple regression

Predictor variables: characteristics of the
studies
Outcome variable: treatment effect estimate
Evidence-based

45

2006-7

Sensitivity analysis
A type of subgroup analysis that considers
non-patient characteristics, e.g., treatment
variations or study methodology
Determines the extent heterogeneity
affected the results of a meta-analysis
If the results are weak, sensitivity analysis
may reveal significant treatment effects when
different methods are used
Evidence-based

46

2006-7

Narrative versus systematic

reviews and meta-analyses
There are no strict rules regarding the
creation of either type
Therefore, it may be difficult to decide if a
given review is systematic or narrative

Narrative reviews do not typically use

systematic methods
They tend to be subjective and prone to bias
Cover broader topics than systematic reviews
Evidence-based

47

2006-7

Narrative vs.
systematic reviews (cont.)
May be unclear how conclusions were
drawn from the data in narrative reviews
Often the number of studies supporting one
side of a topic is counted and then compared
with the number supporting the opposite side
The side with the highest number of
supporting articles wins
This process does not consider the weight of
studies as in systematic reviews
Evidence-based

48

2006-7

Advantages and disadvantages of

narrative versus systematic reviews
Narrative
reviews

Advantages

Disadvantages

Present a general overview

covering a specific topic
which provides primary
information and/or an update
Fairly easy for novice
authors to prepare

May not provide the best

available answers to focused
questions
Findings are less reliable

Systematic Present a comprehensive

reviews
review of the literature based
on all available research with
regard to a focused research
question
Provide an estimate of the
true answer to the research
question
Evidence-based

49

Specialized expertise of
reviewers is required
Involve a formal research
protocol
Findings are only relevant
to a single question

2006-7

QUOROM Statement
QUOROM (Quality of Reporting of Metaanalyses)
The QUOROM Statement was developed
to reduce the potential for reviews to reach
contradictory conclusions
An attempt to ensure uniformity and
accurate reporting
Has been adopted by many journals
Evidence-based

50

2006-7

Structure of review articles

Should conform to the anatomy of a
typical scholarly article
i.e., Abstract, Introduction, Methods, Results,
Discussion, and References

Literature reviews are in reality a type of

research
However, conclusions are derived from
original sources of information
Evidence-based

51

2006-7

Structured Abstract
Objective
The author should clearly state the purpose of
the article

Background
A description of what prompted the review
Presentation of a context for the review

Methods
A description of the methods used
Evidence-based

52

2006-7

Structured Abstract (cont.)

Discussion
The implication and relevance of the
information the review presents

Conclusion
Summary of what the review contributes to
the literature
What new conclusion can be drawn as a
result of the synthesis of the literature
Evidence-based

53

2006-7

Introduction
Presents the background and context of
the problem that inspired review
The topic should be clearly defined, and
tied in with clinical practice
Novel terms should be defined
A description of the course of the disease,
common outcomes and treatment options
A synopsis of existing research
Evidence-based

54

2006-7

Introduction (cont.)
The importance and need for the review
should be established by showing
That there are gaps in the literature on the
topic
The extent of the conditions negative impact
on society in terms of human suffering and
monetary costs
Explained in enough detail to substantiate the
need for the review
Evidence-based

55

2006-7

Introduction (cont.)
A focused and well-constructed question
should be present
Provides direction for the review
Assists readers in determining if the review is
applicable to their individual clinical
circumstances
Should help establish the reviews inclusion
criteria
Evidence-based

56

2006-7

Methods
Describes the search process and
strategies involved, including:
Databases searched
Search terms
Search limits
e.g., publication years, languages, ages, etc.

Should include enough detail to enable

others to replicate the search
Evidence-based

57

2006-7

Methods should include

The criteria that were used to include or
exclude studies
For example, exclude surgery related studies
or drug trials

A description of how studies were

appraised
Rating instruments are typically used
However, the reliability of these instruments
varies considerably
Evidence-based

58

2006-7

Methods should include (cont.)

Information about
How the relevance of primary studies was
ascertained
How the data were extracted and synthesized
Sources of heterogeneity
How much
How it was handled

Evidence-based

59

2006-7

Results
The outcome of the search process is
presented
Including information on
The number of articles retrieved
How many articles were excluded from the
review and which of the inclusion criteria they
failed to meet
Look for evidence of selective referencing
Evidence-based

60

2006-7

Results (cont.)
The Results section is sometimes very
short
When short, details about the retrieved
articles are provided in the Discussion section

However, it may be longer

The characteristics of the included studies
may be described and contrasted in this
section
Often presented in tables
Evidence-based

61

2006-7

Discussion
The findings of all of the articles in the
review are synthesized to generate a
conclusion
There may or may not be a separate
Conclusions section

Information about the etiology,

pathophysiology, diagnosis, treatment,
and prognosis of the condition at issue is
often provided
Evidence-based

62

2006-7

Discussion (cont.)
Presents a new perspective on the topic
that is usually more reliable than any of
the individual articles in the review
Caution authors have the leeway to
defend articles that support their viewpoint
and challenge those that do not
Systematic methods control for much of this
subjectivity, but it is still possible in the best
types of reviews
Evidence-based

63

2006-7

Discussion (cont.)
The synthesis is the heart of the literature
review design
Consequently it is important to ensure that a
meaningful integration is accomplished

The author should offer an interpretation

of the literature reviewed
A critical appraisal of the articles reviewed
may be in order
Evidence-based

64

2006-7

Discussion (cont.)
The conclusion should be in agreement
with the evidence presented in the review
Authors should emphasize what new
information can be gained
The conclusion should not merely repeat
what was previously written

Evidence-based

65

2006-7

References
Should be comprehensive and cite all
articles included in the review
Derived almost entirely from peerreviewed journals
But may include conference proceedings,
textbooks, and government documents
Unpublished works too; but keep in mind, they
have not been peer-reviewed
Evidence-based

66

2006-7

References (cont.)
No padding of references
Only enough articles should be cited to make
a point

Be alert for authors who misrepresent

sources, which is a fairly common
occurrence
e.g., misquote, selectively quote, omit
relevant information, etc.
May have to read source material to detect
Evidence-based

67

2006-7

Appraisal of review articles

Recent reviews are typically more useful
because they contain the latest research
However, even the latest review may be behind
the most current research

The literature search should involve

multiple databases
The search strategy should be clearly
described, along with the criteria used for
study inclusion
Evidence-based

68

2006-7

Appraisal of reviews (cont.)

Was a clear study question asked?
Reviews should have a clear purpose in the
form of a study question or statement of
purpose
Especially with systematic reviews, but also
applies to narrative reviews

Otherwise, the review will lack direction and

may generate faulty conclusions

Evidence-based

69

2006-7

Appraisal of reviews (cont.)

Was the study question focused on a
specific clinical issue?
The study question must be focused on a
specific clinical issue that is relevant to patient
care to help practitioners make clinical
decisions
Broad topic reviews provide background
information, but are not very helpful in
answering clinical questions
Evidence-based

70

2006-7

Appraisal of reviews (cont.)

Were the inclusion/exclusion criteria
that were used to select articles
provided and were they appropriate?
Selection criteria should be appropriate for the
clinical question that prompted the search
Difficult to assess the breadth of the review or
judge the extent of the authors partiality
without this information
Evidence-based

71

2006-7

Appraisal of reviews (cont.)

Was an adequate literature search
conducted?
The search should be capable of finding all
relevant studies
Appropriate databases and search terms
Citations should be collected from the
reference sections of the included articles
How likely is it that relevant studies were
missed during the search?
Evidence-based

72

2006-7

Appraisal of reviews (cont.)

Was the validity of the studies included in
the review assessed?
Reviews that include articles without
appraising them are poorer quality and less
reliable
Authors may examine issues such as the
methods of randomization and blinding,
whether concealment of the allocation to
groups was used, etc.
Evidence-based

73

2006-7

Appraisal of reviews (cont.)

Were the studies appraised in a
dependable manner?
Should be reviewed by more than one
reviewer to strengthen confidence in their
findings
Reliable appraisal instruments should be used
because they yield more consistent results
Often included in an appendix

Evidence-based

74

2006-7

Appraisal of reviews (cont.)

Were the included studies randomized
trials?
Strengthens the usefulness of a literature
review considerably when RCTS are involved
Reviews that include observational studies
provide little evidence to support treatment
However, such reviews are helpful with patient
care and qualify as the best evidence when
observational research is all that exists
Evidence-based

75

2006-7

Appraisal of reviews (cont.)

Were the results of the included studies
similar?
Heterogeneity of studies may occur due to:
Clinical factors (e.g., dissimilar participants,
interventions or outcomes)
Methodological factors (e.g., sample size or
method of randomization)
Contradictory results just by chance

Be concerned when discrepancies are large

Evidence-based

76

2006-7

Appraisal of reviews (cont.)

Do the results of the synthesis logically
flow from the studies that were included?
Authors have a great deal of latitude in the
way they interpret and judge articles
Studies should be weighted based on study
size and validity
Reviews that simply compare the number of
positive findings to the number with negative
findings may be misleading
Evidence-based

77

2006-7

Appraisal of reviews (cont.)

If the review dealt with therapy, was an
estimate of treatment effect provided?
This is the overall estimate of treatment effect
size in a meta-analysis
The width of the associated confidence
interval reveals the precision of the estimate
A wide confidence interval degrades precision
and lowers the degree of certainty regarding
the reviews findings
Evidence-based

78

2006-7

Appraisal of reviews (cont.)

Do the results of the review help with the
care of patients?
In order to effectively use the results of a
review in the care of a specific patient, the
populations should be similar
The outcomes used in the review should be
clinically important
Comparable patient and review interventions
e.g., are manipulation and mobilization equal
Evidence-based

79

2006-7

Appraisal of reviews (cont.)

Were directions for future research
offered?
Especially helpful in reviews because authors
assess the sum of all research on a topic and
are in a very good position to determine what
types of studies should be carried out in the
future

Evidence-based

80

2006-7