Flodgren 2016

Cochrane Database of Systematic Reviews
Tools developed and disseminated by guideline producers to

promote the uptake of their guidelines (Review)
Flodgren G, Hall AM, Goulding L, Eccles MP, Grimshaw JM, Leng GC, Shepperd S
Flodgren G, Hall AM, Goulding L, Eccles MP, Grimshaw JM, Leng GC, Shepperd S.
Tools developed and disseminated by guideline producers to promote the uptake of their guidelines.
Cochrane Database of Systematic Reviews 2016, Issue 8. Art. No.: CD010669.
DOI: 10.1002/14651858.CD010669.pub2.
www.cochranelibrary.com
Tools developed and disseminated by guideline producers to promote the uptake of their guidelines (Review)
Copyright © 2016 The Cochrane Collaboration. Published by John Wiley & Sons, Ltd.
TABLE OF CONTENTS
HEADER . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
ABSTRACT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
PLAIN LANGUAGE SUMMARY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
SUMMARY OF FINDINGS FOR THE MAIN COMPARISON . . . . . . . . . . . . . . . . . . . 4
BACKGROUND . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
OBJECTIVES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
METHODS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
RESULTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
Figure 1. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Figure 2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Figure 3. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
DISCUSSION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
AUTHORS’ CONCLUSIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
ACKNOWLEDGEMENTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
REFERENCES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
CHARACTERISTICS OF STUDIES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
DATA AND ANALYSES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
ADDITIONAL TABLES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
APPENDICES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
CONTRIBUTIONS OF AUTHORS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
DECLARATIONS OF INTEREST . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
SOURCES OF SUPPORT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
DIFFERENCES BETWEEN PROTOCOL AND REVIEW . . . . . . . . . . . . . . . . . . . . . 84
Tools developed and disseminated by guideline producers to promote the uptake of their guidelines (Review) i
[Intervention Review]
Tools developed and disseminated by guideline producers to

promote the uptake of their guidelines
Gerd Flodgren1 , Amanda M Hall2 , Lucy Goulding3 , Martin P Eccles4 , Jeremy M Grimshaw5 , Gillian C Leng6 , Sasha Shepperd7
1 The Norwegian Knowledge Centre for the Health Services, Norwegian Institute of Public Health, Oslo, Norway. 2 Nuffield Department
of Population Health, The George Institute for Global Health, Oxford, UK. 3 King’s Improvement Science, King’s College London,
London, UK. 4 Institute of Health and Society, Newcastle University, Newcastle upon Tyne, UK. 5 Clinical Epidemiology Program,
Ottawa Hospital Research Institute, Ottawa, Canada. 6 National Institute for Health and Care Excellence, London, UK. 7 Nuffield
Department of Population Health, University of Oxford, Oxford, UK
Contact address: Sasha Shepperd, Nuffield Department of Population Health, University of Oxford, Rosemary Rue Building, Old
Road Campus, Headington, Oxford, Oxfordshire, OX3 7LF, UK. [email protected].
Editorial group: Cochrane Effective Practice and Organisation of Care Group.

Publication status and date: New, published in Issue 8, 2016.
Review content assessed as up-to-date: 3 February 2016.
Citation: Flodgren G, Hall AM, Goulding L, Eccles MP, Grimshaw JM, Leng GC, Shepperd S. Tools developed and disseminated
by guideline producers to promote the uptake of their guidelines. Cochrane Database of Systematic Reviews 2016, Issue 8. Art. No.:
CD010669. DOI: 10.1002/14651858.CD010669.pub2.
ABSTRACT
Background
The uptake of clinical practice guidelines (CPGs) is inconsistent, despite their potential to improve the quality of health care and patient
outcomes. Some guideline producers have addressed this problem by developing tools to encourage faster adoption of new guidelines.
This review focuses on the effectiveness of tools developed and disseminated by guideline producers to improve the uptake of their
CPGs.
Objectives
To evaluate the effectiveness of implementation tools developed and disseminated by guideline producers, which accompany or follow
the publication of a CPG, to promote uptake. A secondary objective is to determine which approaches to guideline implementation
are most effective.
Search methods
We searched the Cochrane Effective Practice and Organisation of Care (EPOC) Group Specialised Register, Cochrane Central Register
of Controlled Trials (CENTRAL); NHS Economic Evaluation Database, HTA Database; MEDLINE and MEDLINE In-Process and
other non-indexed citations; Embase; PsycINFO; CINAHL; Dissertations and Theses, ProQuest; Index to Theses; Science Citation
Index Expanded, ISI Web of Knowledge; Conference Proceedings Citation Index - Science, ISI Web of Knowledge; Health Management
Information Consortium (HMIC), and NHS Evidence up to February 2016. We also searched trials registers, reference lists of included
studies and relevant websites.
Selection criteria
We included randomised controlled trials (RCTs) and cluster-RCTs, controlled before-and-after studies (CBAs) and interrupted time
series (ITS) studies evaluating the effects of guideline implementation tools developed by recognised guideline producers to improve
the uptake of their own guidelines. The guideline could target any clinical area.
Tools developed and disseminated by guideline producers to promote the uptake of their guidelines (Review) 1
Data collection and analysis
Two review authors independently extracted data and assessed the risk of bias of each included study using the Cochrane ’Risk of bias’
criteria. We graded our confidence in the evidence using the approach recommended by the GRADE working group. The clinical
conditions targeted and the implementation tools used were too heterogenous to combine data for meta-analysis. We report the median
absolute risk difference (ARD) and interquartile range (IQR) for the main outcome of adherence to guidelines.
Main results
We included four cluster-RCTs that were conducted in the Netherlands, France, the USA and Canada. These studies evaluated the
effects of tools developed by national guideline producers to implement their CPGs. The implementation tools evaluated targeted
healthcare professionals; none targeted healthcare organisations or patients.
One study used two short educational workshops tailored to barriers. In three studies the intervention consisted of the provision of
paper-based educational materials, order forms or reminders, or both. The clinical condition, type of healthcare professional, and
behaviour targeted by the CPG varied across studies.
Two of the four included studies reported data on healthcare professionals’ adherence to guidelines. A guideline tool developed by the
producers of a guideline probably leads to increased adherence to the guidelines; median ARD (IQR) was 0.135 (0.115 and 0.159
for the two studies respectively) at an average four-week follow-up (moderate certainty evidence), which indicates a median 13.5%
greater adherence to guidelines in the intervention group. Providing healthcare professionals with a tool to improve implementation of
a guideline may lead to little or no difference in costs to the health service.
Authors’ conclusions
Implementation tools developed by recognised guideline producers probably lead to improved healthcare professionals’ adherence to
guidelines in the management of non-specific low back pain and ordering thyroid-function tests. There are limited data on the relative
costs of implementing these interventions.There are no studies evaluating the effectiveness of interventions targeting the organisation
of care (e.g. benchmarking tools, costing templates, etc.), or for mass media interventions. We could not draw any conclusions about
our second objective, the comparative effectiveness of implementation tools, due to the small number of studies, the heterogeneity
between interventions, and the clinical conditions that were targeted.
PLAIN LANGUAGE SUMMARY

Effectiveness of tools developed and disseminated by guideline producers to improve uptake of their guidelines
Background
Clinical practice guidelines (CPGs) are evidence-based recommendations for healthcare professionals about the care of patients with
specific conditions. The uptake of CPGs by healthcare professionals is inconsistent, despite their potential to improve the quality of
health care and patient outcomes. Some guideline producers have addressed this problem by developing tools to encourage the adoption
of new guidelines.This review focuses on the effectiveness of tools developed and distributed by recognised guideline producers to
improve the uptake of their CPGs.
Characteristics of included studies
Researchers from Cochrane searched the literature up to February 2016 and identified four randomised studies evaluating the effects
of tools developed by recognised guideline producers to implement their guidelines. These were developed by guideline producers in
France, the Netherlands and in the USA and Canada. In all four studies the interventions targeted the healthcare professional. None of
the tools specifically targeted the organisation of care or the patient. The clinical conditions, and the healthcare professionals’ behaviour
targeted by the CPG, varied across studies, as did the tools used to improve guideline implementation.
Key results
Two of the four included studies reported on how well healthcare professionals stick to guideline recommendations when providing
care to their patients, depending on whether they received a CPG with a tool aimed at improving the use of the CPG, or if they received
the CPG only. The results of this review show that healthcare professionals who received a guideline tool together with the CPG on
the management of non-specific low back pain or ordering thyroid-function tests probably stick more closely to the recommendations,
compared with those who received the CPG only. A guideline tool aimed at improving the use of a guideline, may lead to little or no
difference in cost to the health service.
Certainty of the included evidence
The included evidence was from randomised controlled trials, which is considered the highest level of evidence. However, due to high
risk of bias in the included studies our confidence in the effect on observing guideline recommendations was moderate. Our confidence
in the evidence for cost effectiveness was low, since only a single study provided evidence for this comparison.
S U M M A R Y O F F I N D I N G S F O R T H E M A I N C O M P A R I S O N [Explanation]
Clinical practice guideline (CPG) + implementation tool compared with CPG only for adherence to guidelines
Patient or population: Healthcare prof essionals (physiotherapists, hospital physicians) providing care f or people with
dif f erent clinical conditions (patients with non specif ic low back pain, patients with sym ptom s indicating a need f or a thyroid
f unction test)
Setting: Private physiotherapy clinics in the Netherlands, general hospitals in France
Intervention: CPG + guideline im plem entation tool (e.g. training workshops, paper-based m aterials and order f orm s,
rem inders, web-based tools)
Comparison: CPG only
Outcomes M edian ARD No of Participants Certainty of the evi- Comments

(Absolute risk differ- (studies) dence
ence) (GRADE)
(IQR)
Adherence to guide- Guideline tools pro- 68 physiotherapy ⊕⊕⊕ 2 of the 4 included stud-
lines vided to healthcare pro- practices; and 6 hospi- moderate 1 ies reported a proxy
f essionals as an aid tals m easure f or adherence
to im prove the use (2 C-RCTs) to guidelines, and re-
of a CPG probably sults f rom these stud-
lead to im proved adher- ies could theref ore not
ence with the CPG, as be included in the ARD
com pared to guidelines calculation
only. M edian ARD: 0.
135 (0.115 to 0.15.9) at
m ean 4 weeks f ollow-
up
Costs Guideline tools aim ed 68 physiotherapy clin- ⊕⊕ 1 trial reported no dif -
at im proving the use of ics low2 f erence in m ean direct
guidelines m ay lead to (1 C-RCT) annual cost* per pa-
little or no dif f erence in tient between interven-
healthcare costs tion and control groups.
1 French paper belong-
ing to 1 of the in-
cluded trials (6 hospi-
tals) and reporting on
costs awaits transla-
tion
* Direct costs included costs of the dissem ination of the guideline and healthcare resource use by the patient
GRADE Working Group grades of evidence
High certainty: Further research is very unlikely to change our conf idence in the estim ate of ef f ect.
M oderate certainty: Further research is likely to have an im portant im pact on our conf idence in the estim ate of ef f ect and
m ay change the estim ate.
Low certainty: Further research is very likely to have an im portant im pact on our conf idence in the estim ate of ef f ect and is
likely to change the estim ate.
Very low certainty: We are very uncertain about the estim ate
1
We downgraded the certainty of the evidence one step due to high risk of bias.
2 We downgraded the certainty of the evidence two steps due to im precision.
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
BACKGROUND system). Learning modules are a popular approach to support-

ing the use of CPGs; for example, NICE has developed a range
The uptake of clinical practice guidelines (CPGs) is inconsistent, of online educational tools (NICE 2012b) in collaboration with
despite their potential to improve the quality of health care and BMJ Learning, the Nursing Times and e-Learning for Health (for
patient outcomes (Grol 2003; Schuster 1998; Seddon 2001). A va- example the eVTE online educational tool to reduce the risk of
riety of tools to improve the uptake of CPGs have been developed, venous thromboembolism (eVTE 2013)). The goal is to enable
but not always by the guideline producers themselves. This review users of CPGs to be more aware of recent evidence as summarised
focuses on the effectiveness of tools developed and distributed by in the relevant NICE guidance and to apply the newly acquired
guideline producers to improve the uptake of their CPGs. These knowledge in their practice and address any potential barriers. Ex-
tools are either embedded within a guideline, for example tailoring amples of CPG producers working within health systems to im-
a guideline recommendation for a particular user group, or they prove the uptake of their CPGs include: NICE working within
accompany the CPG, for example interactive learning modules the National Health Service (NHS) in England and Wales by pro-
developed to support the use of a CPG. viding commissioners with quality standards (NICE 2016); the
Scottish Intercollegiate Guidelines Network (SIGN) which pro-
vides problem-based small-group learning modules (SIGN 2012);
Description of the condition the American College of Cardiology providing a guideline clinical
app and running the Guidelines in Practice (GAP) project to pro-
CPGs have the potential to improve healthcare delivery and out- vide customised, guideline-specific implementation tools (ACC
comes, but the adoption of guidelines by healthcare professionals 2016; Mehta 2002); the Veterans Health Administration adapt-
and health system managers is inconsistent, and gaps remain being their CPGs for colorectal screening to local health organisa-
tween recommended care and clinical practice. Previous system- tions; Kaiser Permanente’s healthcare system which has developed
atic reviews have identified a range of interventions to support the and implemented their Proactive Officer Encounter Programme
implementation of guidelines (Grimshaw 2004). However, most to provide clinical decision support to increase the uptake of their
of these have been developed independently of the producers of own and other CPGs (Kanter 2010); and the National Prescrib-
guidelines. In response to this some guideline producers have de- ing Centre in the UK that set up the ’communities of practice’
veloped tools to improve the uptake of their CPGs. Some of these (the NHS Medicines and prescribing communities of practice).
interventions focus on improving the skills needed to apply ev- Data from NHS Evidence show that 92% (33/36) of guideline
idence to practice and others aim to integrate the content of a producers submitting their CPGs for accreditation by NHS Ev-
CPG into a local healthcare system. The value of these tools has idence publish support tools intended to aid implementation of
been questioned by the UK National Institute for Health and Care their guidance (NHS 2012a). Many guideline producers are work-
Excellence (NICE), (New Reference, Leng 2013 [pers comm]) as ing on transforming their narrative CPGs into electronic format,
they are an additional investment for the guideline producer, and as this may improve uptake through the implementation of CPGs
the evidence of the effectiveness of this approach is uncertain. in computer-based decision-support systems (Peleg 2010).
Description of the intervention How the intervention might work

Interventions developed and disseminated by guideline producers Producers of CPGs who develop implementation interventions to
to improve the consistent use of CPGs by health professionals and support their use have focused on the information needs of dif-
health system managers include learning modules (which may be ferent user groups. Interventions are aimed at improving aware-
accredited with Continuing Medical Education (CME) points), ness of CPGs, strengthening the skills needed to understand and
education outreach visits (for example, academic detailing), com- implement a CPG, and supporting the use of a CPG within the
munication tools (for example, press releases following the publi- context of a local healthcare system (Greenhalgh 2005). Tailoring
cation of CPGs) or tailored formatting (for example, the wording the implementation of interventions to facilitate practice change
of recommendations adapted for a target audience or local health (to promote a CPG) typically involves identification of the deter-
minants of healthcare practice. This can include discussions with by users of CPGs, or are associated with their use:
healthcare professionals about potential barriers and systems re- 1. Usability: the structure of the CPG has been modified to
quiring change (Baker 2015), identification of ways to facilitate facilitate access, for example by providing a one-page summary
change and designing, applying and assessing appropriate inter- of the recommendations;
ventions (Wensing 2011). The Guideline Implementability Ap- 2. Adaptability: the CPG is available in different formats for
praisal (GLIA) instrument may be used by producers of guidelines different users or purposes, for example, print and electronic
to identify barriers to implementation during the design phase of format, versions of the CPG are available for patients and
a CPG and enable modifications prior to publication (Shiffman caregivers;
2005). For example, templates may be developed for users of CPGs 3. Validity: using a standardised system to grade the quality of
to populate with local data in order to assess the applicability and evidence supporting each recommendation, for example
impact of a CPG. The tailoring process is also important in en- GRADE;
gaging clinicians in the implementation process (Horbar 2004; 4. Applicability: the wording of the CPG recommendation
Titler 2009). Findings from a recent survey of more than 300 NHS has been tailored for different target audiences to support
commissioning staff, who use CPGs to guide decision making, application of the guidance to local circumstances; this may
confirm the importance of these approaches. Local public health include clinical and contextual information;
intelligence, expert advice and examples of best practice appear to 5. Communicability: information to supplement the CPG,
be the most sought-after types of evidence, and in order for knowl- for example, educational resources for patients and information
edge to be used it has to be translated into a practical resource to support patient involvement;
(Gkeredakis 2011). Finally, if a guideline producer has authority 6. Accommodation: the addition of information on costs and
and works within the health system, or is perceived to be influen- resources, for example, the costing templates provided by NICE,
tial, the uptake of CPGs may be improved (Rogers 1995). and information on competencies and training required to
Other determinants of the effective implementation of all CPGs implement the recommendations;
are that they are clearly written, specific to a population and con- 7. Implementation: information on potential barriers and
text, easy to use and that there is research evidence of its effective- strategies for facilitating implementation, for example, a clinical
ness for a particular end-user’s work context (Titler 2001). Guide- assessment using a point-of-care template;
line development is usually carried out by a multidisciplinary, na- 8. Evaluation: performance measures or quality indicators for
tionally-representative group, who conduct a systematic review to audit and monitoring.
identify and critically appraise the evidence, and who ensure that
the guideline recommendations are explicitly linked to the sup-
porting evidence. Expert opinions are also used in CPGs where Why it is important to do this review
research evidence is not available. Producers of guidelines can also
CPGs can improve healthcare delivery and outcomes, but the
use the AGREE tool by which the quality of a guideline may be
adoption of guidelines by clinicians and healthcare managers is
evaluated, thus allowing end-users to decide how well a guideline
inconsistent. Previous Cochrane Reviews have described the ef-
has been developed and whether it will be applicable to the setting
fectiveness of a range of interventions to support the implemen-
in which they are working (AGREE 2010).
tation of guidelines (Akl 2013; Flodgren 2011; Flodgren 2013a;
The format used to communicate the content of a CPG can also
Forsetlund 2009; Giguère 2012; Grilli 2002; Jamtvedt 2006;
influence its adoption (Greenhalgh 2005; Rogers 1995). While
O’Brien 2007; Shojania 2009). However, most of these have been
CPGs are frequently written as text documents (Peleg 2010), stud-
developed independently of the producers of guidelines. Respond-
ies have shown that clinicians usually do not use written guide-
ing to continued concern about the inconsistent use of CPGs,
lines during the actual care process (Wang 2002). Instead, patient-
some national guideline producers have developed and imple-
specific advice, particularly if delivered during patient encounters,
mented tools to support the uptake of their CPGs. This is an addi-
is suggested to be more effective in changing clinician behaviour
tional investment for the guideline producer and the effectiveness
(Shea 1996). Thus, implementing CPGs in computer-based de-
of this approach is not known. The focus of this review is to assess
cision-support systems may improve the acceptance and appli-
the effectiveness of implementation tools, developed and dissem-
cation of guidelines in daily practice, particularly if the actions
inated by guideline producers, on the uptake of their guidelines.
and observations of healthcare workers are monitored and advice
These tools may require changes to the presentation of the CPG
is generated whenever a guideline is not followed (Wang 2002).
(e.g. tailoring of a CPG), or to be published alongside CPGs (e.g.
One example of guideline producers who have provided healthcare
online learning modules).
professionals with clinical decision support to increase the uptake
of CPGs is the Kaiser Permanente healthcare system with their
Proactive Officer Encounter Programme (Kanter 2010).
Gagliardi 2011 identified eight features of CPGs that are desired
OBJECTIVES
To evaluate the effectiveness of implementation tools developed 1. Tools targeting the healthcare professional
and disseminated by guideline producers, which accompany or
follow the publication of a CPG, to promote uptake.
A secondary objective is to determine which approaches to guide- i) Tailoring

line implementation are most effective.
• Tailoring of CPGs for different users to improve usability
and applicability: examples include using different wording,
varying the content, incorporating case studies of patients’
METHODS experiences in the form of vignettes or narratives which
contextualise the recommendations.
• Different CPG formats adapted for different users/
purposes, e.g. electronic (for use on a Personal Digital Assistant),
Criteria for considering studies for this review paper, multimedia versions, summaries, the inclusion of
algorithms.
Types of studies
ii) Education
We aimed to include randomised controlled trials (RCTs), cluster-
• Learning modules (to include interactive learning modules)
randomised trials (C-RCTs), controlled before-and-after studies
which may be accredited with Continuing Medical Education
(CBAs) and interrupted time series (ITS) studies evaluating the
(CME) points, or to support the use of audit by junior doctors.
effects of guideline implementation tools developed by recognised
• Instructions/templates, e.g. instructions, tools or templates
guideline producers to improve the uptake of their own guide-
to tailor guidelines/recommendations for local context (may also
lines. CBAs were eligible for inclusion if they involved at least two
be used at the organisational level); point-of-care templates/
intervention and two control sites, and ITS studies were eligible
forms (clinical assessment, standard orders).
if they had at least three data points before and three data points
• Decision-support systems, e.g. electronic guidelines with
after the intervention.
built-in decision-support systems.
Types of participants
2. Tools targeting the patient
We included all qualified healthcare professionals, health system
managers and policy makers. • Producing versions of CPG recommendations for the
We excluded studies involving trainees or medical students. public to improve provider-patient communication about
guideline recommendations.
Types of interventions
3. Tools targeting the organisation of care
We included any interventions developed by producers of CPGs to
improve guideline implementation. Guideline producers include, • Benchmarking tools, e.g. measures of gaps in performance
for example, the World Health Organisation (WHO), NICE, and to be used by those monitoring the implementation of CPGs
SIGN. As guidelines may be produced for a specific jurisdiction, (may also be used by individual healthcare professionals).
health system, or group of healthcare professionals, interventions • Costing templates as a budgetary aid (may also be used by
to improve the implementation of these CPGs may be distributed individual healthcare professionals) to assess the resources
to organisations but targeted at individuals within the organisa- required to implement the CPG.
tion, or they may be targeted at entire organisations. We used the • Programme evaluation, audit tools, performance measures
definition of a CPG developed by the USA Institute of Medicine: and quality indicators to evaluate the implementation of the
“clinical guidelines are systematically developed statements to as- CPG.
sist health care professional and patient decisions about appropri-
ate health care for specific clinical circumstances” (Field 1990).
Using the EPOC taxonomy (EPOC taxonomy 2002) as a guide, 4. Mass media interventions
we developed the following classification to organise and define
interventions as below: • Press releases following the publication of a CPG.
The comparisons are as follows: • CINAHL (Cumulative Index to Nursing and Allied Health
1. Tools developed by a guideline producer versus a tool Literature), EbscoHost (1982 to 3 February 2016)
developed by another organisation or a guideline user (i.e. tools • Dissertations and Theses, ProQuest (3 February 2016)
developed independently of the CPG producer). • Index to Theses (up to 3 February 2016)
2. Tools developed by a guideline producer versus no tool (i.e. • Science Citation Index Expanded, ISI Web of Knowledge
CPGs alone). (1945 to 3 February 2016)
We excluded the following types of studies/interventions: • Conference Proceedings Citation Index - Science, ISI Web
1. Tools developed by groups of researchers, guideline groups of Knowledge (1990 to 3 February 2016)
on commission (no longer existing). • Health Management Information Consortium (HMIC),
2. Studies describing tools developed by guideline producers NHS Evidence (1979 to 3 February 2016)
to improve guideline uptake without providing objective The MEDLINE search strategy (Appendix 1) was translated
measurements of the effect of these interventions on professional for other databases using appropriate syntax and vocabulary for
practice or patient outcomes. those databases. The strategy included Medical Subject Headings
3. Surveys of barriers/facilitators to the uptake of guidelines. (MeSH) and synonyms for guidelines and implementation. Re-
sults were limited by two methodological filters: the Cochrane
Types of outcome measures Highly Sensitive Search Strategy (sensitivity- and precision-max-
imising version, 2008 revision) to identify randomised trials, and
We included studies reporting the following outcome measures:
an EPOC methodology filter to identify non-RCT designs. We did
not apply language or publication status restrictions. Search strate-
Main outcomes gies for the other databases are found in Appendix 2; Appendix 3;
Appendix 4; Appendix 5; Appendix 6; Appendix 7; Appendix 8;
Objective measures of healthcare professional performance,
Appendix 9.
healthcare resource use or patient outcomes.
Searching other resources

Secondary outcomes
Self-reported measures of healthcare professional performance and We searched the following additional sources:
healthcare manager performance, including knowledge or use of
CPGs, and costs.
Grey literature
We excluded studies that only included self-reported outcomes.
We conducted a ’grey literature’ search to identify studies not
indexed in the databases listed above. Sources included the sites
listed in Appendix 10. We document guideline websites searched
Search methods for identification of studies
in Appendix 11.
Electronic searches Trial registries

Information specialist Nia Roberts (NR) developed the search We searched the following registries:
strategy for MEDLINE in consultation with the review authors, • International Clinical Trials Registry Platform (ICTRP),
and searched the Cochrane Database of Systematic Reviews and the Word Health Organization (WHO) (www.who.int/ictrp/en/)
Database of Abstracts of Reviews of Effects (DARE) up to February • ClinicalTrials.gov, US National Institutes of Health (NIH)
2016 for related systematic reviews, and the following databases (clinicaltrials.gov/)
for primary studies.
• Cochrane Effective Practice and Organisation of Care We also :
(EPOC) Group Specialised Register, Reference Manager • reviewed reference lists of all included studies, relevant
• Cochrane Central Register of Controlled Trials systematic reviews/primary studies/other publications;
(CENTRAL) (Cochrane Library), Wiley (3 February 2016) • contacted authors of relevant studies or reviews to clarify
• NHS Economic Evaluation Database, HTA Database reported published information/seek unpublished results/data;
(Cochrane Library), Wiley (3 February 2016) • contacted researchers with expertise relevant to the review
• MEDLINE and MEDLINE In-Process and other non- topic/EPOC interventions, as well as guideline-producing bodies
indexed citations, OvidSP (1946 to 3 February 2016) regarding any further published or unpublished research;
• Embase, OvidSP (1974 to 3 February 2016) • conducted cited reference searches for all included studies
• PsycINFO, OVIDSP (1967 to 3 February 2016) in ISI Web of Knowledge.
Data collection and analysis 1. Similar baseline characteristics;
2. Similar baseline outcome measures;
3. Reliable main outcome measures;
Selection of studies 4. Adequate protection against contamination.
We resolved disagreements by discussion among review authors.
We downloaded all titles and abstracts retrieved by electronic
We assigned an overall assessment of the risk of bias (high, mod-
searching to the reference management database Endnote (
erate or low risk of bias) to each of the included studies using the
EndNote X7) and removed duplicates. Four review authors (from
approach suggested in Chapter 8 of the Cochrane Handbook for
GF, AH, LG, SS) and an additional systematic reviewer indepen-
Systematic Reviews of Interventions (Higgins 2011). We considered
dently examined the remaining references, excluded those studies
studies with low risk of bias for all key domains, or where it seems
which clearly did not meet the inclusion criteria, and produced a
unlikely that bias seriously alter the results, to have a low risk of
long list of possible included studies. Two review authors (from GF,
bias. We rated studies as high risk of bias if at least one domain
AH and SS) scrutinised these citations, obtained full-text copies
was unclear or studies were judged to have some bias that could
of potentially relevant references, and independently assessed the
plausibly raise doubts about the conclusions as being at moderate
eligibility of the retrieved full-text papers. We resolved disagree-
risk of bias. We considered studies with a high risk of bias in at
ments by discussion among review authors.
least one domain or judged to have serious bias that decreases the
certainty of the conclusions, to have a high risk of bias.
Data extraction and management
Two review authors (from GF, AH and LG) independently ex- Measures of treatment effect
tracted data from each study into a modified EPOC data extrac-
For each study, we reported data in natural units. Where baseline
tion form. We resolved disagreements by discussion among review
results were available from RCTs we reported pre-intervention and
authors. We extracted the following information: setting; location;
postintervention means or proportions for both study and control
characteristics of healthcare professionals; type of healthcare or-
groups. We also calculated the absolute risk difference (ARD) for
ganisation; intended population of guideline; type and target of
each reported dichotomous outcome, using baseline data when
intervention; the components of the intervention; the compari-
available.
son intervention; any information about the time (and resources)
needed to implement or use the tool, or both; costs and outcomes
reported.
Unit of analysis issues
We also extracted data on any collaborative effort between pro-
ducers and users of guidelines aiming to improve the development There were no unit of analysis issues, all studies adjusted for clus-
of implementation tools, e.g. engagement of individual healthcare tering (Bekkering 2005; Daucourt 2003; Fine 2003; Shah 2014).
professionals or the organisation of care or both in the develop-
ment; assessment of barriers/facilitators to CPG adoption at the
Dealing with missing data
provider level or at the organisational level, or both; or assessment
of the healthcare professionals’ or the organisation of care’s readi- We did not contact authors to request missing data, for example,
ness to change. when the main outcome was graphically presented without nu-
We used the Review Manager 5 software developed by Cochrane merical data.
(Review Manager 2014) to structure the content of the review
when writing it up for publication.
Assessment of heterogeneity
Due to the heterogeneity between studies in terms of populations,
Assessment of risk of bias in included studies clinical conditions/targeted behaviour, and implementation tools
Two review authors (from GF, AH, and LG) independently as- used, meta-analysis was not feasible, and we therefore did not assess
sessed the risk of bias of each included study using the Cochrane’s statistical heterogeneity.
’Risk of bias’ tool (Higgins 2011) on six standard criteria:
1. Adequate sequence generation;
2. Adequate concealment of allocation; Assessment of reporting biases
3. Blinded or objective assessment of main outcome(s); As meta-analysis of main outcomes was not feasible, we did not
4. Adequately addressed incomplete outcome data; assess publication bias through a funnel plot. However, our search
5. Free from selective outcome reporting; for studies to include was extensive and covered a number of guide-
6. Free from other potential sources of bias. line web sites, Guideline Clearing Houses and professional asso-
We used four additional criteria specified by EPOC (EPOC 2015): ciations.
Data synthesis We did not perform a sensitivity analysis, as no meta-analysis was
As we did not find sufficiently homogeneous studies to permit conducted.
meta-analysis, we reported, for dichotomous outcomes, the me-
dian of medians of effect sizes across studies, as has previously been
done in other reviews (Flodgren 2011: Grimshaw 2004; Shojania
2009). When multiple adherence outcomes were reported within RESULTS
individual studies, we first calculated the median effect size (range)
across all outcomes reported in each study, and then calculated the
median of medians and interquartile range (IQR) across studies.
Two review authors used the GRADE tool
Description of studies
(www.gradeworkinggroup.org/) to judge the overall certainty of
the evidence for each outcome, using the following domains: risk
of bias, inconsistency, imprecision, indirectness and publication Results of the search
bias. We downgraded the evidence for serious concerns about each See Characteristics of included studies; Characteristics of ongoing
of these domains. We resolved disagreements through discussions studies and Characteristics of studies awaiting classification tables.
among the review authors. We presented the grading of the evi- The electronic searches yielded 47,181 citations, down to 26,384
dence in Summary of findings for the main comparison. after removal of duplicates. Additional sources searched (includ-
ing websites and reference lists) yielded 473 citations. Of the
these 26,857 citations, we excluded 25,801 irrelevant studies and
Subgroup analysis and investigation of heterogeneity retrieved and scrutinised 1,056 studies. Of these 1,056 double-
As all of the included studies used implementation tools that tar- screened studies we excluded 1030 studies and added 14 to the
geted the healthcare professional, and only one study targeted both excluded studies table. We listed one study protocol and one
the healthcare professional and the patient, we did not undertake conference abstract under ’Ongoing studies’ (Salbach 2014; Te
any subgroup analyses. Boveldt 2011), and two studies under ’Studies awaiting classifica-
tion’ (Maximov 2012; Van Driel 2007). We judged four studies to
be eligible for inclusion in the review. See study flowchart Figure
Sensitivity analysis
1.
Figure 1. Study flow diagram.
oral antibiotics for people with pneumonia (Fine 2003); and im-
Included studies
proved cardiovascular risk screening and risk reduction in people
We identified four eligible studies of cluster-RCTs (Bekkering with diabetes (Shah 2014).
2005; Daucourt 2003; Fine 2003; Shah 2014) for inclusion in The guideline recommendations (n = 4) that were implemented
this review, of which one (Shah 2014) consisted of two separate were described in one of the studies (Bekkering 2005).
cluster-RCTs: one a population-based C-RCT including all family
practices in Ontario, Canada, and the other an embedded C-RCT
including a subsample of the family practices from the larger study. Guideline producers
See Table 1 for details on the guideline development process.
In Bekkering 2005 the Royal Dutch Physiotherapy Association
Populations developed the guidelines; in Daucourt 2003 the Committee
for Co-ordinating Clinical Evaluation and Quality in Aquitaine
(CCECQA) developed the guidelines, together with regional
Healthcare professionals groups and national guideline developers; in Fine 2003 mem-
In Bekkering 2005 the participants were physiotherapists (n = bers of the Pneumonia Patient Outcomes Research Team (PORT)
113); and in two studies (Daucourt 2003; Fine 2003) the par- project developed the guidelines; and in Shah 2014 the Canadian
ticipants were physicians other than general practitioners (GPs) Diabetes Association (CDA) developed the guidelines.
(n = 1913), or family physicians (number not reported), and in
one study the intervention was targeted at family physicians (Shah
Description of the intervention
2014). None of the studies targeted patients, health system man-
agers or policy makers. See Table 2 , and Table 3.
Patients i) Interventions targeting the healthcare professional
The number and clinical condition of participants in the included All four studies evaluated guideline implementation tools targeting
studies were as follows: participants (n = 500) with non-specific the healthcare professional.
low back pain (Bekkering 2005); participants (n = 608) with hos-
pital-acquired pneumonia (Fine 2003); an unknown number of
patients who required a thyroid-function test (Daucourt 2003); Tailored interventions
and people with diabetes > 40 years old (n = 933,769) in Ontario
Bekkering 2005 assessed the effectiveness of two (2½ hours) ed-
(administrative study) and a subgroup of people with diabetes (n
ucational training sessions for groups of 8 to 12 physiotherapists
= 1592) at high risk of cardiovascular disease (clinical study) (Shah
on adherence to CPGs for management of non-specific low back
2014).
pain. The sessions were based on interventions reported as being
effective in the literature (e.g. interactive education and discussion,
feedback, and reminders) and were tailored to barriers found in a
Settings
survey.
Bekkering 2005 was set in private physiotherapy practices (n = 68);
two studies (Daucourt 2003; Fine 2003) were set in hospitals (n =
13), and Shah 2014 was set in family practices (n = 4007 and n =
80 respectively). The studies were conducted in the Netherlands Printed materials
(Bekkering 2005), France (Daucourt 2003), the USA (Fine 2003) Three studies evaluated the effectiveness of paper-based educa-
and in Canada (Shah 2014) . tional materials or reminders, or both (Daucourt 2003; Fine 2003;
Shah 2014).
Daucourt 2003 evaluated the combined effect of two tools: a mem-
Targeted behaviour orandum pocket card and a test request form to implement guide-
The clinical conditions/behaviours targeted by the CPG were as lines for appropriate thyroid-test ordering. Orders were made by
follows: care for people with non-specific low back pain (Bekkering checking a box, with boxes corresponding to inappropriate test
2005); appropriate thyroid-test ordering (Daucourt 2003); timely ordering shaded and therefore making ordering impossible.The
conversion (and discharge) from intravenous antibiotic therapy to physician could overrule this by writing the order at the bottom
of the sheet. The pocket card summarised the recommendations for changing professionals’ behaviour and systematic reviews on
according to the various clinical or therapeutic situations requiring the effectiveness of implementation interventions was also used to
a thyroid test. determine the content of the implementation strategy.
In Fine 2003 physicians received a multifaceted guideline inter-
vention which included placement of a detail sheet in the patient’s
medical record once a patient met guideline criteria for stability
when receiving intravenous antibiotic therapy for pneumonia, a Theory base of interventions
follow-up recommendation to the attending physician, and an of- None of the interventions used in the included studies was theory-
fer to arrange follow-up home nursing care. The three site-specific based.
detail sheets promoted any of three recommended action(s), i.e.
conversion from intravenous to oral antibiotic therapy only, con-
version and hospital discharge, or hospital discharge only.
Evidence base of interventions
Shah 2014 used a cardiovascular disease toolkit which was a col-
lection of printed educational materials, packaged in a brightly- The implementation strategies used in the included studies were
coloured box with CDA branding, sent to Canadian family physi- all supported by some evidence of their effectiveness and cited
cians. The contents included an introductory letter from the Chair high-quality Cochrane Reviews, systematic reviews or overviews
of the practice guidelines’ Dissemination and Implementation to justify their choice of strategies.
Committee; an eight-page summary of selected sections of the
practice guidelines targeted towards family physicians; a four-page
synopsis of the key guideline elements pertaining to cardiovascular
Fidelity
disease risk; a small double-sided laminated card with a simplified
algorithm for cardiovascular risk assessment, vascular protection None of the included studies provided information on interven-
strategies, and screening for cardiovascular disease; and a pad of tion fidelity.
tear-off sheets for patients with a cardiovascular risk self-assess-
ment tool and a list of recommended risk reduction strategies.
Delivery of the intervention
The median duration that an intervention was delivered was 22
weeks (range 4 weeks to 12 months).
ii) Interventions targeting the patient Mode of delivery:

None of the included studies evaluated interventions that targeted In Bekkering 2005 the intervention was delivered face-to-face. In
the patient. two studies (Daucourt 2003; Shah 2014) the paper-based inter-
ventions were provided passively. In Fine 2003 one part of the
intervention was delivered over the phone, and the rest passively
iii) Interventions targeting the organisation of care in the form of paper-based materials.
None of the included studies evaluated interventions that exclu-
sively targeted the organisation of care.
Provider delivering the intervention (if not electronic, paper-
iv) Interventions targeting the healthcare professionals and based, etc):
the patients In Bekkering 2005 the principal investigator and two additional
None of the included studies evaluated targeted both healthcare trainers delivered the intervention. In Fine 2003 a nurse delivered
professionals and patients. part of the intervention.
Comparison interventions
Assessment of barriers The comparison intervention in all included studies was passive
In one of the three included studies (Bekkering 2005), barriers to guideline dissemination. Additional material that was delivered
guideline implementation were assessed through the means of a together with the guideline was as follows: in Bekkering 2005 four
survey to inform the shape and content (i.e. tailoring) of the guide- forms: a self-evaluation form to assess whether their current man-
line implementation strategies. Another aim of the survey was to agement was consistent with the recommendations contained in
retrieve information on the most important discrepancies between the clinical guidelines, two forms facilitating discussion with other
current practice and recommendations of the guidelines. A model physiotherapists and general practitioners respectively, a copy of
the Quebec Back Pain Disability Scale, and a summary of the mean annual cost per patient, total cost for releasing the guideline
CPG. In Fine 2003 a cover letter was sent signed by the hospital’s and cost of active implementation intervention (Bekkering 2005).
utilisation management director describing the rationale for the The other article awaits translation (Saillour-Glénisson 2005).
guideline. In Daucourt 2003 all physicians were invited to a local
information meeting. In Shah 2014 control participants received
the CDA newsletter, which included the revised guideline. Patient outcomes
Bekkering 2005 reported quality-of-life measures at four weeks.
Fine 2003 reported all-cause and pneumonia-related mortality,
Outcomes
medical complications, functional status and patient satisfaction
with care at 30 days after the initial hospitalisation. Shah 2014
reported (primary outcome in administrative data study) death
Healthcare professional outcomes
or non-fatal myocardial infarction. Daucourt 2003 reported the
Two of the four included studies reported a measure of healthcare number of requests for a thyroid function test that complied with
professional adherence to guidelines (Bekkering 2005; Daucourt the guidelines (Guideline Conformity Rate (GCR)) at 4 weeks.
2003) at four weeks; these were included in the calculations of the (See Table 4 for details of the secondary outcomes reported).
median absolute risk difference (ARD).
Excluded studies
Healthcare resource use and costs
After scrutinising the full text we excluded 1030 studies and added
Fine 2003 reported length of initial hospital stay and re-admissions 14 to the excluded studies table. See Characteristics of excluded
at 30 days after index hospitalisation. Shah 2014 reported (primary studies table.
outcome in clinical study) the proportion of patients with diabetes
at high risk of a cardiovascular event who were prescribed a statin
(see Table 4 for details on secondary outcomes reported).
Risk of bias in included studies
Two studies reported on costs (Bekkering 2005; Saillour-Glénisson See ’Risk of bias’ tables within the Characteristics of included
2005 (belonging to Daucourt 2003)). One of the studies reported studies, Figure 2 and Figure 3.
Figure 2. Risk of bias summary: review authors’ judgements about each risk of bias item for each included
study. White space indicate studies not reporting non-objective outcomes and for which risk of bias could not
be assessed.
Figure 3. Risk of bias graph: review authors’ judgements about each risk of bias item presented as
percentages across all included studies. White spcace indicate studies not reporting non-objective outcomes
and for which risk of bias could not be assessed.
The randomisation sequence and the allocation concealment were

adequate in three studies (Bekkering 2005; Daucourt 2003; Shah Effects of interventions
2014), and unclear in Fine 2003. In Shah 2014 blinding was ad- See: Summary of findings for the main comparison
equate (clinical data study assessed), and in Daucourt 2003 it was
unclear whether or not the healthcare professionals were blinded, i) Interventions targeting the health care professionals
while in two studies (Bekkering 2005; Fine 2003) it was clear that
they were not. In one study the healthcare professional selected a
maximum of 10 consecutive patients for the study, and we there- Healthcare professional outcomes
fore judged the risk of performance bias to be high (Bekkering See Summary of findings for the main comparison; Table 4 and
2005). Performance bias was also judged high in Fine 2003 as Table 5.
treatment assignment was not concealed. Blinding of outcome Two of the four included studies (Bekkering 2005; Daucourt
assessment was adequate in two studies (Bekkering 2005; Shah 2003) reported one or more measures of healthcare professionals’
2014), and unclear in the other two. Baseline characteristics were adherence to guidelines. The overall median absolute risk differ-
reported to be similar in one study (Fine 2003), not similar in ence (range) (five comparisons) was (range: 0.115 to 0.159), i.e. a
one study (Shah 2014), and unclear in the other two studies. The median difference in adherence of 13.5%, with the effects ranging
outcome data were complete in two studies (Daucourt 2003; Shah from 11.5% to 15.9% increase in adherence.
2014), and unclear in the other two (with losses to follow-up of Fine 2003, in which physicians received an educational mailing,
more than 20%). In Shah 2014 some of the outcomes that were a daily assessment of (pneumonia) patient stability and an addi-
listed in the trial protocol were not in the study report, while in tional sheet to the medical notes with a follow-up recommenda-
the other three studies the risk of selective reporting was low. Shah tion for converting from intravenous to oral antibiotic and hospital
2014 had unclear risk of other bias (contamination), while the discharge, compared with education mailing alone, reported that
other three were at low risk. those in the intervention group had a more rapid rate of conver-
sion to oral antibiotics (hazard ratio (HR) 1.23, 95% confidence
interval (CI) 1.00 to 1.52, P = 0.06). Shah 2014 did not report iv) Interventions targeting the healthcare professional, the
health professional outcomes. organisation of care and/or the patient
No study reported results for this comparison.
Healthcare resource use and costs

Effectiveness of different approaches of guideline
Fine 2003 reported similar percentages in each group of patients
dissemination
being readmitted (intervention group 14% versus 11% in the con-
trol group), and a similar length of initial hospital stay (median of We include four studies in this review, of which one evaluated the
five days in each group) at 30 days after index visit. effectiveness of two short tailored educational workshops, and the
Shah 2014 reported similar or slightly lower (= undesired effect) other three studied the effects of using paper-based tools, includ-
use of different types of coronary artery disease (CAD) assessment ing order forms or reminders, or both. As the types of multifaceted
tools in practices that received the guideline tool compared to interventions, the clinical condition and behaviour targeted varied
those who received the updated guideline only (administrative across studies it was not possible to determine which of the dif-
data study), as was the case for the medication initiation outcomes ferent approaches used to improve implementation of guidelines
(both were secondary outcomes). was most effective.
Bekkering 2005 reported mean annual direct medical costs for the
intervention group of EUR 374 versus EUR 449 in the control
group. Direct costs included costs of the dissemination of the
guideline and healthcare resource use by the patient. Daucourt DISCUSSION
2003 reported prescribing cost data in a paper in French (Saillour-
Glénisson 2001) which awaits translation.
Summary of main results
Patient outcomes We identified four eligible cluster-RCTs for inclusion in this re-
view, evaluating the effects of tools developed by existing guideline
See Table 4 for details producers to improve implementation of their guidelines.
Bekkering 2005 reported similar quality-of-life scores for patients All included studies evaluated tools that targeted the healthcare
with non-specific low back pain at 12 months. professional. However, meta-analysis was not feasible, since the
Fine 2003 reported similar scores on the SF-12 physical compo- targeted clinical conditions and behaviour, as well as the guideline
nent score (intervention group 45 (standard deviation (SD) 7) tools used, all varied between studies. The variation in the duration
versus control group 45 (SD 7)) and the mental component score of interventions and follow-up also made comparisons difficult.
(intervention group 45 (SD 6) versus control group 45 (SD 7)) at Tools developed by guideline producers, and given to healthcare
30 days after index stay, and little or no difference for mortality professionals as an aid to improve compliance, probably lead to
(intervention group 8% versus control group 9%), and return to greater adherence to guidelines (median absolute risk difference
work (HR 0.99, 95% CI 0.63 to 1.58). The same authors reported (ARD) 13.5%) at an average four weeks follow-up (moderate-
fewer hospital complications in the intervention group compared quality evidence). The effect ranged from 11.5% in one study (two
with control (157 (55%) and 206 (63%) respectively, P = 0.04). tailored short educational workshops to improve management of
Shah 2014 reported little or no difference between groups (Inter- non-specific low back pain) to 15.9% in the other (a pocket mem-
vention 2.5%; Control 2.5%; odds ratio (OR) 1.00, 95% CI 0.96 orandum card and test-request form to improve thyroid-test order-
to 1.03, P = 0.77) for death and non-fatal myocardial infarction ing). Neither study reported baseline adherence, and it appeared
(primary outcome in the administrative data study), and also little that no guideline for the specific targeted behaviours and condi-
or no difference for any of the other (secondary) clinical events tions was previously in place. There was low certainty of evidence
reported (see Table 4 for details). from one trial for little or no difference in costs between groups.
Due to the few eligible studies identified, and the variety of inter-
ventions implemented, we could not determine which approaches
ii) Interventions targeting the organisation of care are most effective, which was the secondary objective of this re-
No studies reported results for this comparison. view. Two of the included studies reported on cost data, and one
of these awaits translation. While it is not possible to directly ad-
dress the investment made by guideline producers in developing
implementation tools, the cost is not likely to differ substantially
iii) Interventions targeting the patient
from other organisations that develop tools to support the im-
No studies reported results for this comparison. plementation of guidelines. It should be noted that even small to
moderate intervention effects may be highly cost-effective if the We are not aware of any other reviews that have evaluated the
targeted clinical condition is highly prevalent and the implemen- effectiveness of tools developed by recognised guideline produc-
tation tools used are inexpensive to develop and to disseminate. ers to improve implementation of their own CPGs. However, our
There is no evidence available for the effectiveness of interventions results of a median 13.5% greater adherence to guidelines in the
targeting the organisation of care or the patient. intervention group (two studies: one evaluating a paper-based in-
tervention, and one an intervention consisting of two short ed-
ucational workshops) are greater than the reported median ab-
Overall completeness and applicability of solute improvement in performance for point-of-care computer
evidence reminders of around 4% (Shojania 2009), 2% for printed edu-
cational materials (Giguère 2012) and 6% for educational meet-
In all included studies the interventions targeted the healthcare ings (Forsetlund 2009). These reviews, however, included a much
professional. None of the included studies used tools that targeted larger number of studies and participants, which may explain the
health system managers or policy makers, the patient (e.g. versions differences in effect.
of the guideline developed for the patient), or the organisation of
care (e.g. benchmarking tools, costing templates or programme
evaluation, audit tools, performance measures and quality indica-
tors to evaluate the implementation of the CPG), and no study
AUTHORS’ CONCLUSIONS
evaluated the effects of mass media interventions. The implemen-
tation tools used were delivered alongside the CPG, and none was
Implications for practice
imbedded within the CPG (e.g. tailoring of the CPG for a specific
audience). In addition, only guideline tools to promote the use There is a range of guideline tools that guideline producers could
of CPGs for a few clinical conditions and behaviours have been develop. However, for tools developed by large guideline-produc-
evaluated. ing bodies, there is limited evidence about their effectiveness. It is
difficult to draw robust conclusions about the tools evaluated in
our review, given the small number of studies and heterogeneity
in study conditions, interventions, and outcomes.
Quality of the evidence
The evidence was from cluster-RCTs that had all taken clustering Implications for research
into account in the analysis. We downgraded all included studies
Given that many CPG developers are providing tools to support
from high to moderate certainty of evidence for the main outcome
implementation, they should consider embedding rigorous evalu-
(adherence to guidelines), due to high risk of bias. As only a single
ations of the tools (e.g. randomised trials) to advance knowledge
study provided evidence for the effectiveness of a certain imple-
in this area. They should also aim to include economic analyses to
mentation on costs, our confidence in the evidence was further
determine the cost effectiveness of their tools.
downgraded to low due to imprecision.
Future studies in this area should also aim to:
Potential biases in the review process • study the effect of organisational interventions, patient
interventions, and of tools embedded in a guideline (e.g.
We searched a large number of databases using a strategy that was tailoring of the content to specific audiences) using a randomised
designed by a senior information specialist, and then adapted for comparison
different databases. We also searched a large number of websites of
relevant guideline producers. Four review authors sifted a number • use validated objective measures of adherence to guidelines
of references identified by the electronic searches, excluding papers and longer follow-up
that were irrelevant and clearly not eligible, and producing a long • report numerical data to accompany graphical figures
list for a second review author to go through. Two review authors
independently assessed all potentially eligible titles and abstracts • describe the process of developing the guideline and the
against the eligibility criteria to ensure no important references implementation tool, including any theory used, the evidence
were missed. We also performed data extraction and assessment of base for the intervention
risk of bias in duplicate. • provide information on who developed the guidelines, and
the guideline development process, as well as describing the
number and complexity of the guideline recommendations
Agreements and disagreements with other • provide information on who delivered the intervention (the
studies or reviews study authors, independent personnel, etc.), and detailed
information on the intensity of interventions (number of face-to- acknowledge Hannah Parke (from King’s improvement Science)
face contacts, contact time, etc.) to permit replication and for assistance with the sifting.
comparison with other studies
Lucy Goulding and Hannah Parke are staff members at King’s Im-
• use the TIDieR checklist (Hoffman 2014) to improve the provement Science, and they wish to acknowledge their funders:
reporting of the characteristics of an intervention King’s Improvement Science is part of the NIHR CLAHRC South
London and comprises a specialist team of improvement scien-
• conduct an economic evaluation, taking into account the
tists and senior researchers based at King’s College London. Its
development of the guideline, and the dissemination and
work is funded by King’s Health Partners (Guy’s and St Thomas’
implementation of the guideline.
NHS Foundation Trust, King’s College Hospital NHS Founda-
tion Trust, King’s College London and South London and Maud-
.
sley NHS Foundation Trust), Guy’s and St Thomas’ Charity, the
Maudsley Charity and the Health Foundation.
This project was supported by the National Institute for Health
Research, via Cochrane Infrastructure funding and a Cochrane
ACKNOWLEDGEMENTS
programme grant to the EPOC Group. The views and opinions
We wish to thank senior information scientist Nia Roberts at the expressed therein are those of the authors and do not necessarily
Bodleian library (Oxford, UK), for developing the search strategy reflect those of the Systematic Reviews Programme, NIHR, NHS
and for running the electronic searches. We would also like to or the Department of Health.
REFERENCES
References to studies included in this review Shah 2014 {published data only}
∗
Shah BR, Bhattacharyya O, Yu CH, Mamdani MM,
Parsons JA, Straus SE, et al. Effect of an educational
Bekkering 2005 {published data only} toolkit on quality of care: a pragmatic cluster randomized
∗
Bekkering GE, Hendriks HJM, Van Tulder MW, Knol trial. PLoS Medicine 2014;11(44):1001588. [10.137/
DL, Hoeijenbos M, Oostendorp RAB, et al. Effect on the journal.pmed.1001588. eCollection 2014]
process of care of an active strategy to implement clinical Shah BR, Bhattacharyya O, Yu CH, Mamdani MM, Parsons
guidelines on physiotherapy for low back pain: a cluster JA, Straus SE, et al. Evaluation of a toolkit to improve
randomised controlled trial. Quality & Safety in Health Care cardiovascular disease screening and treatment for people
2005;14:107–12. with type 2 diabetes: protocol for a cluster-randomised
Hoeijenbos M, Bekkering T, Lamers L, Hendriks E, Van pragmatic trial. Trials 2010; Vol. 11, issue 44:1–7.
Tulder M, Koopmanschap M. Cost-effectiveness of an
active implementation strategy for the Dutch physiotherapy
guideline for low back pain. Health Policy 2005;75(1):
References to studies excluded from this review
85–98.
Daucourt 2003 {published data only} Allegranzi 2013 {published data only}
∗
∗
Daucourt V, Saillour-Glenisson F, Michel P, Jutand M-A, Allegranzi B, Gayet-Ageron A, Damani N, Bengaly L,
Abouelfath A. A multicenter cluster randomized controlled McLaws ML, Moro ML, et al. Global implementation of
trial of strategies to improve thyroid function testing. WHO’s multimodal strategy for improvement of hand
Medical Care 2003;41:432–41. hygiene: a quasi-experimental study. Lancet Infectious
Saillour-Glenisson F, Michel P, Daucourt V. Medico- Disease 2013;13(10):843–51.
Economic assessment of two methods for implementing Pittet D, Allegranzi B, Storr J. The WHO Clean Care
thyroid-testing guidelines. Revue d’Épidémiol et de Santé is Safer Care programme: field-testing to enhance
Publique 2005;53(1):S79–88. sustainability and spread of hand hygiene improvement.
Journal of Infection and Public Health 2008;1(1):4–10.
Fine 2003 {published data only}
Fine MJ, Stone RA, Lave JR, Hough LJ, Obrosky SD, Baker 2001 {published data only}
Mor MK, et al. Implementation of an evidence-based Baker R, Reddish S, Robertson N, Hearnshaw H, Jones
guideline to reduce duration of Intravenous antibiotic B. Randomised controlled trial of tailored strategies to
therapy and length of stay for patients hospitalized with implement guidelines for the management of patients with
community-acquired pneumonia: A randomized controlled depression in general practice. British Journal of General
trial. American Journal of Medicine 2003;115:343–51. Practice 2001;51(470):737–41.
Bosch 2014 {published data only} guidelines on management of asthma and angina in adults
Bosch M, McKenzie JE, Mortimer D, Tavender EJ, Francis in primary care: cluster randomised controlled trial. BMJ
JJ, Brennan SE, et al. Implementing evidence-based 2002;325(7370):941.
recommended practices for the management of patients Eccles M, Rousseau N, Adams P, Thomas L, North of
with mild traumatic brain injuries in Australian emergency England Stable Angina Guideline Development Group.
care departments: study protocol for a cluster randomised Evidencebased guideline for the primary care management
controlled trial. Trials [Electronic Resource] 2014; Vol. 15: of stable angina. Family Practice 2001;18(2):217–22.
281. Eccles M, Rousseau N, Higgins B, Thomas L. Evidencebased
Chan 2013 {published data only} guideline on the primary care management of asthma.
∗
Chan S, Pielak K, McIntyre C, Deeter B, Taddio A. Family Practice 2001;18(2):223–9.
Implementation of a new clinical practice guideline Eccles 2007 {published data only}
regarding pain management during childhood vaccine Eccles MP, Steen IN, Whitty PM, Hall L. Is untargeted
injections. Paediatrics & Child Health 2013;18(7):367–72. educational outreach visiting delivered by pharmaceutical
Taddio A, Appleton M, Bortolussi R, Chambers C, Dubey advisers effective in primary care? A pragmatic randomized
V, Halperin S, et al. Reducing the pain of childhood controlled trial. Implementation Science 2007;2(23):1–8.
vaccination: an evidence-based clinical practice guideline..
Canadian Medical Association Journal 2010;182(18): Flottorp 2002 {published data only}
∗
E843–55. Flottorp S, Oxman AD, Håvelsrud K, Treweek S, Herrin J.
Taddio A, Ipp M, Apppleton M, Chambers C, Halperin Cluster randomised controlled trial of tailored interventions
S, Lockett D, et al. Implementing best practices for to improve the management of urinary tract infections in
vaccination pain management. Pediatrics and Child Health women and sore throat. BMJ 2002;325(7360):1–6.
[Canada] Conference: 91st Annual Conference of the Flottorp S, Oxman AD, Håvelsrud K, Treweek S, Herrin J.
Canadian Paediatric Society Montreal, QC Canada.. 2014; Identifying barriers and tailoring interventions to improve
Vol. 19(6):e102–3, issue 6:e102–3. the management of urinary tract infections and sore throat:
Taddio A, Shah V, Wang J, Parikh C, Smart S, Ipp M, et al. apragmatic study using qualitative methods. BMJ 2003;325
Usability and knowledge testing of educational tools about (7360):1–10.
infant vaccination pain management directed to postnatal Fretheim 2006 {published data only}
nurses. BMC Medical Education 2015;15(45):1–8. Fretheim A, Aaserud M, Oxman AD. Rational Prescribing
De Beurs 2015 {published data only} in Primary Care (RaPP):Economic evaluation of an
∗
De Beurs DP, De Groot MH, De Keijser J, Mokkenstorm intervention to improve professional practice. PLoS
J, Van Duijn E, De Winter RFP. The effect of an e-learning Medicine 2006;3(6):e212.
supported Train-the-Trainer programme on implementation Fretheim A, Håvelsrud K, Oxman AD. Rational Prescribing
of suicide guidelines in mental healthcare. Journal of in Primary care (RaPP): process evaluation of an intervention
Affective Disorders 2015;175:446–53. to improve prescribing of antihypertensive and cholesterol-
De Beurs DP, Groot MH, Bosmans JE, Keijser J, lowering drugs. Implementation Science 2006;1(19):1–9.
∗
Mokkenstorm J, Verwey B, et al. Reducing patients’ Fretheim A, Oxman AD, Håvelsrud K, Treweek
suicide ideation through training mental health teams in S, Kristoffersen DT. Rational Prescribing in Primary
the application of the Dutch multidisciplinary practice Care (RaPP):A Cluster Randomized Trial of a Tailored
guideline on assessment and treatment of suicidal behavior: Intervention. PLoS Medicine 2006;3(6):e134. [DOI:
Study protocol of a randomized controlled trial. Trials 10.1371/journal.pmed.0030134]
2013; Vol. 14:1–10. [DOI: 10.1186/1745-6215-14-372] Fretheim A, Oxman AD, Treweek S, Bjørndal A. Rational
De Beurs DP, Groot MH, Keijser J, Verwey B, Mokkenstorm Prescribing in Primary Care (RaPP-trial). A randomised
J, Twisk JW, et al. Improving the application of a practice trial of a tailored intervention to improve prescribing of
guideline for the assessment and treatment of suicidal antihypertensive and cholesterol-lowering drugs in general
behavior by training the full staff of psychiatric departments practice [ISRCTN48751230]. BMC Health Service Research
via an e-learning supported Train-the-Trainer program: 2003;3(5):1–8.
study protocol for a randomized controlled trial. Trials Overbeek 2010 {published data only}
2013; Vol. 14:1–10. [DOI: 10.1186/1745-6215-14-9] Overbeek LI, Hermens RP, Van Krieken JH, Adang EM,
De Groot M, De Beurs DP, De Keijser J, Kerkhof FJM. Casparie M, Nagengast FM, et al. Electronic reminders
An e-learning supported Train-the-Trainer program for pathologists promote recognition of patients at risk
to implement a suicide practice guideline. Rationale, for Lynch syndrome: cluster-randomised controlled
content and dissemination in Dutch mental health care. trial. Virchows Archiv 2010;456:653–9. [DOI: 10.1007/
Internet Interventions 2015;2:323–9. [DOI: 10.1186/ s00428-010-0907-7]
1745-6215-14-9] Robling 2002 {published data only}
Eccles 2002 {published data only} Robling MR, Houston HLA, Kinnersley P, Hourihan MD,
∗
Eccles M, McColl E, Steen N, Rousseau N, Grimshaw Cohen DR, Hale J, et al. General practitioners’ use of
J, Parkin D, et al. Effect of computerised evidence based magnetic resonance imaging: An open randomised trial
comparing telephone and written requests and an open resources/features/guideline-clinical-app 2016 (accessed
randomised controlled trial of different methods of local August 2016).
guideline dissemination. Clinical Radiology 2002;57:402–7. AGREE 2010
Rood 2005 {published data only} Brouwers M, Kho ME, Browman GP, Burgers JS, Cluzeau
Rood E, Bosman RJ, Van der Spoel JI, Taylor P, Zandstra F, Feder G, et al. AGREE II: Advancing guideline
DF. Use of a computerized guideline for glucose regulation development, reporting and evaluation in healthcare.
in the intensive care unit improved both guideline adherence Canadian Medical Association Journal 2010;E839-842:18-
and glucose regulation. Journal of the American Medical 23.
Informatics Association 2005;12(2):172–80. Akl 2013
Rycroft-Malone 2012 {published data only} Akl EA, Kairouz VF, Sackett KM, Erdley WS, Mustafa
Rycroft-Malone J, Seers K, Crichton N, Chandler J, RA, Fiander M, et al. Educational games for health
Hawkes CA, Allen C, et al. A pragmatic cluster randomised professionals. Cochrane Database of Systematic Reviews 2013,
trial evaluating three implementation interventions. Issue 3. [DOI: 10.1002/14651858.CD006411.pub3]
Implementation Science 2012;7:80. Baker 2015
Baker R, Camosso-Stefinovic J, Gillies C, Shaw EJ,
Witt 2004 {published data only}
Cheater F, Flottorp S, et al. Tailored interventions to
Witt K, Knudsen E, Ditlevsen S, Hollnagel H. Academic
address determinants of practice. Cochrane Database
detailing has no effect on prescribing of asthma medication
of Systematic Reviews 2015, Issue 4. [DOI: 10.1002/
in Danish general practice: a 3-year randomized controlled
14651858.CD005470.pub3]
trial with 12-monthly follow-ups. Family Practice 2004;21
(3):248–53. Beaulieu 2004
Beaulieu M-D, Brophy J, Jacques A, Blais R, Battista R,
References to studies awaiting assessment et al. Drug treatment of stable angina pectoris and mass
dissemination of therapeutic guidelines: a randomized
Maximov 2012 {published data only} controlled trial. QJM 2004;97(1):21–31.
Maximov D, Lesnyak O. Implementation of the Bebb 2007
osteoarthritis clinical guideline: Results of a cluster Bebb C, Kendrick D, Coupland C, Madeley R, Stewart J,
randomized trial in primary care. Annals of Rheumatic Brown K, et al. A cluster randomised controlled trial of the
Disorders 2012; Vol. 71, issue Suppl 3:307. effect of a treatment algorithm for hypertension in patients
Van Driel 2007 {published data only} with type 2 diabetes. British Journal of General Practice
Van Driel ML, Coenen S, Dirven K, Lobbestael J, Janssens 2007;57(535):136–43.
I, Van Royen P, et al. What is the role of quality circles in Bekkering 2003
strategies to optimise antibiotic prescribing? A pragmatic Bekkering GE, Hendriks HJM, Koes BW. Dutch
cluster-randomised controlled trial in primary care. Quality physiotherapy guidelines for low back pain.. Physiotherapy
& Safety in Health Care 2007;16(3):197–202. 2003;89:82–96.
Bero 1998
References to ongoing studies
Bero LA, Grilli R, Grimshaw J, Harvey E, Oxman AD,
Thomson MA. Closing the gap between research and
Salbach 2014 {published data only}
practice: an overview of systematic reviews of interventions
Salbach NM, Wood-Dauphinee S, Desrosiers J, Eng J,
to promote the implementation of research findings. BMJ
Graham I, Jaglal S. Effectiveness of a multi-component
1998;317(7156):465–8.
knowledge translation intervention in improving
Davis 1995
implementation of a stroke rehabilitation guideline: a
cluster randomized controlled trial. Canadian Stroke Davis DA, Thomson MA, Oxman AD, Haynes RB.
Changing physician performance. A systematic review of
Congress Abstracts 2014; Vol. 37, issue Suppl 1:43.
the effect of continuing medical education strategies. JAMA
Te Boveldt 2011 {published data only} 1995;274(9):700–5.
Te Boveldt N, Engels Y, Besse K, Vissers K, Vernooij-Dassen
Davis 1997
M. Rationale, design, and implementation protocol of the
Davis DA, Taylor-Vaisey A. Translating guidelines into
Dutch clinical practice guideline pain in patients with
practice. A systematic review of theoretic concepts, practical
cancer: a cluster randomised controlled trial with Short
experience and research evidence in the adoption of clinical
Message Service (SMS) and Interactive Voice Response
practice guidelines.. Canadian Medical Association Journal
(IVR). Implementation Science 2011;6(126):1–11.
1997;157:408–16.
Additional references eGuidelines
eGuidelines.co.uk. Handbook of United Kingdom and
ACC 2016 European clinical guidelines for primary and shared care.
American College of Cardiology. Guideline Clinical App. www.eguidelines.co.uk/new˙guidelines.php (accessed
http://www.acc.org/tools-and-practice-support/mobile- October 2011).
EndNote X7 [Computer program] Grilli 2002
Thomson Reuters. EndNote X7. Thomson Reuters, 2014. Grilli R, Ramsay C, Minozzi S. Mass media interventions:
EPOC 2015 effects on health services utilisation. Cochrane Database
Effective Practice, Organisation of Care (EPOC). Suggested of Systematic Reviews 2002, Issue 1. [DOI: 10.1002/
risk of bias criteria for EPOC reviews. epoc.cochrane.org/ 14651858.CD000389]
epoc-specific-resources-review-authors 2015 (accessed 7 Grimshaw 1993
August 2016). Grimshaw JM, Russell IT. Effect of clinical guidelines
eVTE 2013 on medical practice: a systematic review of rigorous
e-Learning for Healthcare. e-VTE. www.e-lfh.org.uk/ evaluations. Lancet 1993;342(8883):1317–22.
programmes/vte/ (accessed 7 August 2016). Grimshaw 1995
Field 1990 Grimshaw J, Eccles M, Russell I. Developing clinically valid
Field MJ, Lohr KN (editors). Clinical Practice Guidelines practice guidelines. Journal of Evaluation in Clinical Practice
- directions for a new programme . Institute of Medicine. 1995;1(1):37–48.
Washington DC: National Academy Press, 1990. Grimshaw 2004
Grimshaw JM, Thomas RE, MacLennan G, Fraser C,
Flodgren 2011
Ramsay CR, Vale L, et al. Effectiveness and efficiency of
Flodgren G, Parmelli E, Doumit G, Gattellari M, O’Brien
guideline dissemination and implementation strategies.
MA, Grimshaw J, et al. Local opinion leaders: effects on
NHS R&D HTA programme 2004.
professional practice and health care outcomes. Cochrane
Database of Systematic Reviews 2011, Issue 8. [DOI: Grimshaw 2006
10.1002/14651858.CD000125.pub4] Grimshaw J, Eccles M, Thomas R, MacLennan G, Ramsay
Flodgren 2013a C, et al. Towards evidence-based quality improvement:
Flodgren G, Conterno LO, Mayhew A, Omar O, Pereira evidence (and its limitation) of the effectiveness of guideline
CR, Shepperd S. Interventions to improve professional dissemination and implementation strategies 1966-1998.
adherence to guidelines for prevention of device-related Journal of General Internal Medicine 2006;21:S14–S20.
infections. Cochrane Database of Systematic Reviews 2013, Grol 2003
Issue 3. [DOI: 10.1002/14651858.CD006559.pub2] Grol R, Grimshaw J. From best evidence to best practice:
Forsetlund 2009 effective implementation of change in patients’ care. Lancet
Forsetlund L, Bjørndal A, Rashidian A, Jamtvedt G, O’Brien 2003;362(9391):1225–30.
MA, Wolf F, et al. Continuing education meetings and Higgins 2011
workshops : effects on professional practice and health care Higgins JPT, Green S (editors). Cochrane Handbook
outcomes. Cochrane Database of Systematic Reviews 2009, for Systematic Reviews of Interventions Version 5.1.0
Issue 2. [DOI: 10.1002/14651858.CD003030.pub2] [updated March 2011]. The Cochrane Collaboration,
2011. Available from www.cochrane-handbook.org.
Gagliardi 2011
Gagliardi AR, Brouwers MC, Palda VA, Lemieux- Hoeijenbos 2005
Charles L, Grimshaw JM. How can we improve guideline Hoeijenbos M, Bekkering T, Lamers L, Hendriks E, Van
use? A conceptual framework of implementability. Tulder M, Koopmanschap M. Cost-effectiveness of an
Implementation Science 2011;6(26):1–11. [DOI: 10.1186/ active implementation strategy for the Dutch physiotherapy
1748-5908-6-26] guideline for low back pain. Health Policy 2005;75(1):
Giguère 2012 85–98.
Giguère A, Légaré F, Grimshaw J, Turcotte S, Fiander M, Hoffman 2014
Grudniewicz A, et al. Printed educational materials: effects Hoffmann TC, Glasziou PP, Boutron I, Milne R, Perera R,
on professional practice and healthcare outcomes. Cochrane Moher D, et al. Better reporting of interventions: template
Database of Systematic Reviews 2012, Issue 10. [DOI: for intervention description and replication (TIDieR)
10.1002/14651858.CD004398.pub3] checklist and guide. BMJ 2014;348:g1687.
Gkeredakis 2011 Horbar 2004
Gkeredakis E, Swan J, Powell J, Nicolini D, Scarbrough H, Horbar JD, Carpenter JH, Buzas J, Soll RF, Suresh G,
Roginski C, et al. Mind the gap: Understanding utilisation Bracken MB, et al. Collaborative quality improvement to
of evidence and policy in health care management practice. promote evidence based surfactant for preterm infants: a
Journal of Health Organization and Management 2011;25 cluster randomised trial. BMJ 2004;329(7473):1–7.
(3):298–314. Jamtvedt 2006
Greenhalgh 2005 Ivers N, Jamtvedt G, Flottorp S, Young JM, Odgaar-
Greenhalgh T, Robert G, Bate P, Macfarlane F, Kyriakidou Jensen J, French SD, et al. Audit and feedback: effects on
O. Diffusion of Innovations in Health Service Organisations. professional practice and healthcare outcomes. Cochrane
A systematic literature review. Oxford: Blackwell Publishing, Database of Systematic Reviews 2012, Issue 6. [DOI:
2005. 10.1002/14651858.CD000259.pub3]
Kanter 2010 Rogers 1995
Kanter M, Martinez O, Lindsay G, Andrews K, Denver Rogers E. Diffusion of Innovation. 5th Edition. New York:
C. Proactive office encounter: A systematic approach to Free Press. A division of Simon & Schuster, 1995.
preventive and chronic care at every patient encounter. Saillour Glénisson 2001
Permanente Journal 2010;14(3):38–43. Saillour-Glénisson F. Guidelines for thyroid function tests
Khunti 1998 in adults. Presse Medical 2001;30:686–692.
Khunti K, Baker R, Robertson N. Development of evidence- Saillour-Glénisson 2005
based review criteria for the management of patients with Saillour-Glénisson F, Michel P, Daucourt V. Medico-
depression in general practice. Primary Care Psychiatry economic assessment of two methods for implementing
1998;4:29–33. thyroid-testing guidelines. Revue d’Épidémiol et de Santé
Leng 2013 [pers comm] Publique 2005;53(1):1S79–88.
Leng G. (National Institute for Health and Care Excellence Schuster 1998
(NICE), UK). [personal communication]. Conversation Schuster MA, McGlynn EA, Brook RH. How good is
with: S Shepperd, (Tools to promote the uptake of the quality of health care in the United States?. Milbank
guidelines Review Study Team, Oxford, UK) 2013. Quarterly 1998;76(4):517-63, 509.
Mehta 2002 Seddon 2001
Mehta RH, Montoye CK, Gallogly M, Baker P, Blount A, Seddon ME, Marshall MN, Campbell SM, Roland MO.
Faul J, et al. GAP Steering Committee of the American Systematic review of studies of quality of clinical care in
College of Cardiology. Improving quality of care for acute general practice in the UK, Australia and New Zealand.
myocardial infarction: The Guidelines Applied in Practice Quality in Health Care 2001;10(3):152–8.
(GAP) Initiative. JAMA 2002;287(10):1269–76. Shea 1996
Murrey 1992 Shea S, Du Mouchel W, Bahamonde L. A meta analysis
Murrey KO, Gottlieb LK, Schoenbaum SC. Implementing of 16 randomized controlled trials to evaluate computer-
clinical guidelines: a quality management approach to based clinical reminder systems for preventative care in
reminder systems. QRB Quality Review Bulletin 1992;16 the ambulatory setting. Journal of the American Medical
(12):423–433. Informatics Association 1996;3(6):399–409.
NHS 2012a Shiffman 2005
National Health Service (NHS). NHS Evidence. Shiffman RN, Dixon J, Brandt C, Essaihi A, Hsiao A,
www.evidence.nhs.uk/accreditation# (accessed 30 March Michel G, et al. The GuideLine Implementability Appraisal
2013). (GLIA): development of an instrument to identify obstacles
to guideline implementation. BMC Medical Informatics and
NICE 2012b
Decision Making 2005;5:23.
National Institute of Clinical Excellence. Online
educational tools. http://www.nice.org.uk/usingguidance/ Shojania 2009
education/educational˙tools.jsp (accessed 30 March 2012). Shojania KG, Jennings A, Mayhew A, Ramsay CR, Eccles
MP, Grimshaw J. The effects of on-screen, point of care
NICE 2016
computer reminders on processes and outcomes of care.
National Institute of Clinical Excellence (NICE). Guidance
Cochrane Database of Systematic Reviews 2009, Issue 3.
and advice list. https://www.nice.org.uk/guidance/
[DOI: 10.1002/14651858.CD001096.pub2]
published?type=qs (accessed 12 August 2016).
SIGN 2012
O’Brien 2007 Scottish Intercollegiate Guidelines Network (SIGN). SIGN
O’Brien MA, Rogers S, Jamtvedt G, Oxman AD, Odgaard- implementation support. www.sign.ac.uk/implementation/
Jensen J, Kristoffersen DT, et al. Educational outreach index.html (accessed 30 March 2012).
visits: effects on professional practice and health care
Titler 2001
outcomes. Cochrane Database of Systematic Reviews 2007,
Titler M, Everett LQ. Translating research into practice.
Issue 4. [DOI: 10.1002/14651858.CD000409.pub2]
Considerations for critical care investigators. Critical Care
Peleg 2010 Nursing Clinics North America 2001;13(4):587–604.
Peleg M, Garber JR. Extending the GuideLine Titler 2009
Implementability Appraisal (GLIA) instrument to identify Titler MG, Herr K, Brooks JM, Xie X-J, Ardery G, Schilling
problems in control flow. AMIA 2010 Symposium ML, et al. Translating research into practice intervention
Proceedings. 2010:627–31. improves management of acute pain in older hip fracture
Review Manager 2014 [Computer program] patients. Health Research and Educational Trust 2009;44(1):
The Nordic Cochrane Centre, The Cochrane Collaboration. 264–87.
Review Manager (RevMan). Version 5.3. Copenhagen: Wang 2002
The Nordic Cochrane Centre, The Cochrane Collaboration, Wang D, Peleg M, Tu SW, Boxwala AA, Greenes RA,
2014. Patel VL, et al. Representation primitives, process models
and patient data in computer-interpretable clinical practice Wensing 2011
guidelines: a literature review of guideline representation Wensing M, Oxman A, Baker R, Godycki-Cwirko M,
models. International Journal of Medical Informatics 2002; Flottorp S, Szecsenyi J, et al. Tailored Implementation
68(1-3):59–70. For Chronic Diseases (TICD): a project protocol.
Implementation Science 2011;6(103):1–8.
Weingarten 2000
Weingarten S. Translating practice guidelines into patient References to other published versions of this review
care: guidelines at the bedside. Chest. 2000;188 (Suppl):4S-
7S 2000;188:Suppl:4S–7S. Flodgren 2013b
Flodgren G, Eccles MP, Grimshaw J, Leng GC, Shepperd S.
Wensing 1998 Tools developed and disseminated by guideline producers to
Wensing M, Van der Weijden T, Grol R. Implementing promote the uptake of their guidelines. Cochrane Database
guidelines and innovations in general practice: which of Systematic Reviews 2013, Issue 8. [DOI: 10.1002/
interventions are effective?. British Journal of General 14651858.CD010669]
Practitioners 1998;48(427):991–7. ∗
Indicates the major publication for the study
CHARACTERISTICS OF STUDIES
Characteristics of included studies [ordered by study ID]
Bekkering 2005
Methods Study design: Cluster-RCT

Unit of allocation: The physiotherapy practice
Guideline development: The guidelines used the Dutch method of developing physio-
therapy guidelines, and evidence from systematic reviews was sought and used as the basis
for the recommendations. If no evidence was available, consensus between experts was
obtained.The guidelines were pilot-tested among 100 physiotherapists and reviewed by
an external multidisciplinary panel. A survey, to assess barriers, was part of the guideline
development process
Guideline implementation tool development: The face-to-face training sessions were
based on interventions that have all been shown to be effective (see below). The content
of the strategy was determined on the basis of information about the expected barriers for
implementation gathered during the development of the clinical guidelines. Two experts
gave advice on the content of the strategy
Theories used: The authors did not reporting using theory to guide the development
of the intervention; they based their intervention on implementation methods known
to be effective (interactive education and discussion, feedback, and reminders)
Sample size calculation: The calculation of sample size was based on a difference of 20%
in adherence between the 2 groups, which was considered to be an important difference.
It was adjusted for the effect of clustering using an ICC of 0.057 and an estimated cluster
size of 5 patients per practice. In total, a sample of 284 patients or 48 practices or both
were needed (2-sided α = 0.05, β = 0.20)
Participants Participating providers: Physiotherapists n = 68 practices (113 physiotherapists); In-

tervention: n = 34 practices (52 physiotherapists); Control: n = 34 practices (61 phys-
iotherapists); 325/6261 = 5.2% of all eligible practices were selected to be invited to
participate, of which 257 practices declined participation (79.1%)
Losses to follow-up and withdrawals: 6 physiotherapists (4 from the intervention
group and 2 from the control group) dropped out immediately after randomisation;
these were more often working in a solo/duo practice (P = 0.038). 9 physiotherapists
from the intervention group did not complete training, and 11 participants also from
intervention group (3 did not complete registration, 8 did not include any patients) and
11 participants from the control group (who did not include any patients) were lost to
follow-up
Characteristics of healthcare professionals:
Mean (SD) experience (years): Intervention: 15.7 (8.8); Control: 14.1 (8.3)
n (%) postgraduate education on low back pain: Intervention: 36 (75.0%); Control: 41
(69.5%)
n (%) postgraduate education on chronic pain: Intervention: 0 (0%); Control: 4 (6.8%)
Patients:patients (n = 500) with non-specific low back pain
Setting: private physiotherapy practices; n =113 physiotherapists.
Location (rural/urban): Central part of the Netherlands
Country: The Netherlands
Bekkering 2005 (Continued)
Interventions Aims: To evaluate the effect on the process of care of an active strategy to implement
clinical guidelines on physiotherapy for low back pain
Type of intervention: Intervention targeting the healthcare professional (educational
intervention/tailoring)
Description of guideline tool: An active strategy to implement the CPGs which con-
sisted of 2 training sessions, each lasting 2½ hours, for groups of 8 - 12 physiothera-
pists. For each session a preparation time of 2 hours was recommended. The sessions
were based on interventions shown to be effective, such as interactive education and
discussion, feedback and reminders.The content of the strategy was determined on the
basis of information about the expected barriers for implementation gathered during
the development of the clinical guidelines.Two experts gave advice on the content of the
strategy
Guideline developers: Royal Dutch Physiotherapy Association (National Physiotherapy
guidelines)
Delivery: Postal delivery of guideline; small-group face-to-face training and reminders;
By whom: The primary investigator and 1 of 2 additional trainers with adequate clinical
experience in the management of low back pain supervised the training sessions
Timing: The guideline was published in 2001, and the study was conducted between
May 2001 and December 2002
Duration of intervention: 2 X 2½ hours (+ 2 hours recommended preparation time),
4 weeks between the first and the second session
Control: All physiotherapists received the clinical guidelines via the standard method
of dissemination (by mail) used by the Royal Dutch Society for Physiotherapy. They
received the guidelines by mail together with 4 forms: a self-evaluation form to assess
whether their current management was consistent with the recommendations contained
in the clinical guidelines, 2 forms facilitating discussion with other physiotherapists and
general practitioners respectively, and a copy of the Quebec Back Pain Disability Scale. A
summary of the clinical guidelines was also provided. At the same time an article about
the development of the guidelines was published in a Dutch professional journal for
physiotherapists
Outcomes Main outcome:

• Adherence to the guidelines
Secondary outcome:
• Costs (reported in Hoeijenbos 2005)
• Quality of life (assessed with the EQ-05 and reported in Hoeijenbos 2005)
Follow-up: 4 weeks after randomisation (adherence outcomes), 12 months (cost out-
comes)
Notes Ethical approval and informed consent obtained (yes/no): Yes

Conflict of interest: None declared
Funding: the Ministry of Health, Welfare and Sports
Risk of bias
Bias Authors’ judgement Support for judgement
Random sequence generation (selection Low risk p. 108, Col. 1, Para. 2

bias) ”Block randomisation (blocks of four prac-
tices) was carried out after pre-stratification
for the work setting (solo/duo practices ver-
sus group practices). A statistician, who was
not involved in this trial, drew up an allo-
cation schedule using a computerised ran-
dom number generator.“
Allocation concealment (selection bias) Low risk The primary investigator (GEB), without
any knowledge of the practices, listed the
practices alphabetically according to the
name of their street address, and subse-
quently assigned them to the intervention
or control group using the allocation sched-
ule
Blinding of participants and personnel High risk Outcome group: adherence to guidelines
(performance bias) The participating physiotherapists could
All outcomes not be blinded to the intervention. The
physiotherapist selected a maximum of 10
consecutive patients for the study. High risk
for performance bias
Blinding of outcome assessment (detection Low risk p.108, Col.1, Para.5

bias) “Two reviewers independently assessed the
All outcomes registration form using the algorithm with-
out being aware of the group allocation. In
total, four reviewers assessed the forms. Be-
fore the final scoring five cases were used for
a pilot assessment and these were blinded
again afterwards. In case of disagreement
between the two reviewers, a method was
used to discuss and resolve the disagree-
ment by consensus. If the disagreement
persisted, a third reviewer made the final
decision.”
Baseline characteristics similar Unclear risk Physiotherapists in the intervention group

were slightly older (P = 0.011), but there
were no other differences between the 2
groups. 500 patients were included
Baseline outcome measures similar Unclear risk The intervention group had a higher qual-
ity-of-life score, 0.6730 (SD 0.2042) com-
pared with the control group 0.6134 (SD
0.2661)
Incomplete outcome data (attrition bias) High risk 37/52 intervention physiotherapists (71.
All outcomes 2%) and 48/61 control physiotherapists
(78.7%) remained in the study at follow-
up
Selective reporting (reporting bias) Low risk Results reported for all outcomes listed in
the Methods section
Other bias Low risk No evidence of other risk of bias
Daucourt 2003

Unit of allocation: The wards
Guideline development: The CCECQA established the guidelines in collaboration
with a regional working group and a national review group. The method combined a
comprehensive review of the literature and expert consensus
Guideline implementation tool development: No information
Theories used: No information
Sample size calculation: The expected prevalence used for the sample size calculation
was the rate of test ordering of “TSH only.” With a probability of 0.05, an error of 0.20,
an expected rate of test ordering of “TSH only” in the control group of 0.50, a minimal
increase in test ordering of the “TSH only” rate of 0.10, an ICC of 0.25 and an estimated
average number of thyroid function tests by cluster (ward) of 40
Participants Participating healthcare professionals: Physicians; n = 704; Intervention (Dual Inter-

vention Group): n = 346; Control Group (guideline only) n = 358). Note: 2 study arms
(Order Form Group, n = 339; Pocket Card Group, n = 369) were not included in this
review
Ward specialty
Medicine : Dual intervention: 63; Control: 76; Emergency: Dual intervention: 2; Con-
trol: 0; Psychiatry: Dual intervention: 34; Control: 24
Surgeon: Dual intervention: 1; Control: 0
Prescriber status
Senior: Dual intervention: 53; Control: 57; Junior: Dual intervention: 43; Control: 40;
Unknown: Dual intervention: 4; Control: 3
Indication of test ordering*
Test of thyroid dysfunction: Dual intervention: 61; Control (guideline only): 59
Therapeutic tests: Dual intervention: 23; Control (guideline only): 23
Other pathologic test: Dual intervention: 16; Control (guideline only): 18
Losses to follow-up and withdrawals: 52 tests were not accounted for
Patients: Patients with symptoms indicating a need for a thyroid function test
Setting: 6 volunteer general hospitals all receiving residents: 2 middle-sized hospitals in
the second and third largest towns in Aquitaine (Pau hospital (535 beds) and Bayonne
hospital (494 beds)), 2 small-sized hospitals (Bergerac hospital (171 beds) and Bouscat
hospital (90 beds)) and 2 psychiatric hospitals (Charles Perrens hospital (904 beds) and
Cadillac hospital (541 beds))
Location (rural/urban): Aquitaine, in South-West France
Daucourt 2003 (Continued)
Country: France
Interventions Aims: To compare the (independent) and combined effectiveness of 2 implementation

interventions (a memorandum pocket card and a test request form) of guidelines for
ordering thyroid function tests
Type of intervention: interventions targeting the healthcare professional
Type of guideline tool: a Memorandum Pocket Card (MPC) and a Test Request Form
(TRF).The TRF replaced the former order sheet. It was a 2-by-2 grid with coloured
boxes (white, grey, black). Orders were made by checking the box at the intersection
between test and clinical situations. Boxes corresponding to inappropriate test ordering
were shaded, therefore making ordering impossible.The physician could overrule this by
writing down the order at the bottom of the sheet. The MPC summarised the recom-
mendations according to the various clinical or therapeutic situations requiring thyroid
exploration. It was small enough for physicians to keep it in their coat pocket and to
consult it before prescribing thyroid function test
Guideline developers: The CCECQA established such guidelines in collaboration with
a regional working group and a national review group
Delivery: Paper-based interventions (and face-to-face meeting)
Timing: Unclear
Duration of intervention: 4 weeks
Control: Physicians in all groups received guidelines and were invited to a local infor-
mation meeting
Outcomes Main outcome:

• Proportion of thyroid function test ordering in accordance with the guidelines
(Guideline Conformity Rate (GCR))
Follow-up: 4 weeks after guideline implementation
Notes Ethical approval and informed consent obtained (yes/no): No information

Conflict of interest: No information
Funding: Supported in part by the Agence Nationale de l’Accreditation et de l’Evaluation
en Santé (ANAES)
Risk of bias
Random sequence generation (selection Low risk p. 433, Col. 2, Para. 1

bias) “Randomization was performed by the
CCECQA using a random number table.”
Allocation concealment (selection bias) Low risk Cluster-RCT .
Blinding of participants and personnel Unclear risk Outcome group: proportion of thyroid-
(performance bias) function test ordering in accordance with
All outcomes the guidelines
It was not explicitly stated if the health-
care professionals ordering the tests were
blinded to the intervention
Daucourt 2003 (Continued)
Blinding of outcome assessment (detection Unclear risk p. 433, Col. 2, Para. 3

bias) “A standardized collection grid was
All outcomes prospectively filled in by a research assistant
in each hospital for all consecutive thyroid
function tests ordered during the collection
period. No information on blinding.”
Baseline characteristics similar Unclear risk Gender differed according to the interven-
tion groups: the proportion of women was
65% in the dual intervention group, 63%
in the order-form group, 73% in the pocket
card group and 61% in the control group
(P < 0.01). The mean patient age was 67
years (SD 20 years) in the dual interven-
tion group, 64 years (SD 20 years) in the
order-form group, 70 years (SD 21 years)
in the pocket card group, and 66 years (SD
17 years) in the control group (P < 0.01)
. No ward/healthcare professional charac-
teristics provided
Baseline outcome measures similar Unclear risk No baseline measure of outcome
Incomplete outcome data (attrition bias) Low risk Indication of test ordering unknown for 52
All outcomes (3.1%) patients (total n = 1464)
Selective reporting (reporting bias) Low risk No evidence of selective outcome reporting
Fine 2003

Unit of allocation: Groups of physicians
Guideline development:The medical practice guideline developed for this project was
based on a review of the evidence of the time to reach clinical stability, and consensus of
an 8-member national guideline panel. The guideline was reviewed by clinical opinion
leaders at each study site, and was approved for local use by the relevant utilisation
management department. The final guideline consisted of explicit clinical criteria to
define stability for conversion from intravenous to oral antibiotic therapy and for hospital
discharge
Guideline implementation tool development: No information
Theories used: No information
Sample size calculation:This study was designed with 80% power to detect a 1-day
decrease in length of stay from an assumed baseline of 7.2 days. The sample size was
adjusted for the clustering on physician group (22), assuming an average of 3.5 patients
per group and an ICC of 0.1
Participants Participating healthcare professionals: 116 groups of physicians who were likely to
treat patients with community-acquired pneumonia: Intervention: 277 physicians (57
groups); Control: 268 physicians (59 groups)
Characteristics of healthcare professionals:
Age (years): Intervention: 47 ± 11; Control: 46 ± 11, P = 0.35
Female: Intervention: 45 (18); Control: 57 (24) , P = 0.09
Medical specialty, P = 0.14
Generalists: Intervention: 190 (73); Control: 192 (79)
Pulmonary specialist: Intervention: 19 (7); Control: 19 (8)
Other specialists: Intervention: 50 (19); Control: 31 (13)
Patients: Patients treated by a participating physician and who had a documented treat-
ment plan for hospital-acquired pneumonia, and a chest radiograph report consistent
with a new pulmonary infiltrate; Intervention: n = 283; Control: n = 325. Note: only
40% of eligible patients were enrolled
Setting: 7 non-profit hospitals: 1 university teaching hospital (site A); 3 community
teaching hospitals (sites B,C and D); 3 community non-teaching hospitals (sites E.F and
G)
Location (rural/urban): Pittsburg, Pennsylvania
Country: USA
Interventions Aims: To determine whether implementation of an evidence-based guideline would

reduce the duration of intravenous antibiotic therapy and length of stay for patients
hospitalised with pneumonia
Type of intervention: Education (detail sheet with treatment recommendations)
Type of guideline tool: An educational mailing delivered to physicians and a daily as-
sessment of patient stability that was coupled with a multifaceted strategy to implement
the project guideline once a patient met criteria for stability. A detail sheet was placed
in the patient’s medical record once a patient met guideline criteria for stability, a fol-
low-up recommendation to the attending physician, and an offer to arrange follow-up
home nursing care. One of the 3 site-specific detail sheets promoting the recommended
action(s) (i.e. conversion from intravenous to oral antibiotic therapy only, conversion
and hospital discharge, or hospital discharge only) was placed in the physician progress
notes section of each patient’s chart immediately following the determination of the
Fine 2003 (Continued)
corresponding type(s) of stability. At this time, the research nurse telephoned or directly
approached the patient’s attending physician to state that the patient met guideline cri-
teria for conversion to oral antibiotic therapy or hospital discharge (or both); to indicate
that the detail sheet had been placed in the medical record and review its content with
the physician; and to offer to take a verbal order for oral antibiotic therapy and make
arrangements for home nursing care
Guideline developers: Researchers who were part of the PORT group
Delivery: Paper-based detail sheets/treatment recommendations; nurse telephone re-
minder
Timing: Once the patient treated with intravenous antibiotics had been deemed to be
in a stable condition according to the guidelines, the intervention tool was delivered.
The CPG was delivered as part of the educational mailing 1 month before recruitment
of patients started
Duration of intervention: 12 months (patients were recruited between 1 February 1998
and 31 March 1999)
Control: The educational mailing was delivered to physicians in both study arms dur-
ing the month before patient recruitment began. The control group receive a practice
guideline alone. This mailing included a cover letter signed by the hospital’s utilisation
management director describing the rationale for the guideline and a written version of
the guideline
Outcomes Main outcomes:

• Duration of intravenous antibiotic therapy
• Length of index hospital stay
• Time to stability (for conversion to oral antibiotics and for discharge)
Secondary outcomes:
• All-cause mortality (data retrieved from records)
• Pneumonia-related mortality (data retrieved from registers)
• Medical complications (data retrieved from medical records)
• Rehospitalisation rates (interview assessed)
• Functional status (subgroup only, results not included in this review)
• Time to return to usual activities (subgroup only, results not included in this
review)
• Patient satisfaction with care
Follow-up: secondary outcomes were assessed 30 days after the index hospitalisation
Notes Ethical approval and informed consent obtained (yes/no): Yes

Conflict of interest: None declared
Funding: The Agency for Healthcare Research and Quality, Rockville, Maryland, and
the National Institute of Allergy and Infectious Diseases (HS08282), Bethesda, Mary-
land. Dr Fine was also supported in part as a Robert Wood Johnson Foundation Gener-
alist Physician Faculty Scholar and by a Career Development award from the National
Institute of Allergy and Infectious Diseases
Risk of bias
Fine 2003 (Continued)
Random sequence generation (selection Unclear risk No information

bias)
Allocation concealment (selection bias) Unclear risk The groups of physicians that were ran-
domised to intervention and control group
were at the same location
Blinding of participants and personnel High risk Outcome group: duration of intravenous
(performance bias) antibiotic therapy
All outcomes “ Because of the nature of the intervention,
physicians and research nurses could not
be blinded to their treatment assignments.
Patients were not informed of their physi-
cians’ treatment assignment”
Blinding of participants and personnel Unclear risk Outcome group: hospitalisations, func-
(non-objective outcomes) tional status
At the 30-day telephone interview, pa-
tients or their proxy respondents were
queried about subsequent hospitalisations
(patient self-report). Functional status was
reassessed with the SF-12 (18) for patient
respondents only
Blinding of outcome assessment (detection Unclear risk Outcome group: duration of intravenous
bias) antibiotic therapy; length of stay for the
All outcomes index hospitalisation. Data retrieved from
registers, but unclear by whom
Baseline characteristics similar Low risk Baseline characteristics similar (Table 2)
Baseline outcome measures similar Unclear risk No baseline measures of outcome
Incomplete outcome data (attrition bias) Unclear risk There were 25 post-enrolment exclusions
All outcomes (providers) in each study arm, and 10 in-
hospital study withdrawals (4 intervention
and 6 control)
Selective reporting (reporting bias) Low risk Results reported for all outcomes listed in
the Methods section
Shah 2014
Methods Study design: 2 separate studies: 1 a population-based multicentre cluster-RCT, and the
other an embedded C-RCT including a subgroup of these practices
Unit of allocation: family practices
Guideline development: Candadian Diabetes Association (CDA) updated their 2008
guideline: reviewed the literature and graded the evidence as well as the applicability of
evidence, and subjected the revised draft guideline to external peer review
Guideline implementation tool development: The CDA formed a Dissemination and
Implementation Committee to create a guideline implementation strategy. The first
component of this strategy was aimed at improving adherence with the recommendations
for cardiovascular disease screening and treatment for people with diabetes. The strategy
highlighted the identification of diabetic patients at high risk for cardiovascular events,
treatment targets and methods for vascular protection, and the selection of patients and
methods for coronary artery disease screening. The toolkit was created for the CDA
by clinical experts including family physicians, endocrinologists, and other healthcare
professionals, with guidance from clinicians with expertise in knowledge translation and
implementation
Theories used: the toolkit was developed without a specific quality improvement or
educational theory to guide its content or delivery
Sample size calculation: Administrative data study: an administrative data base of the
entire population aged ≥ 40 years with diagnosed diabetes in Ontario, which was more
than 900,000 people; the study had .95% power to detect an unadjusted absolute differ-
ence of at least 0.4% in a dichotomous primary outcome, using an α-error of 0.05. Power
was reduced after adjustment for baseline differences and for clustering, but remained
sufficient to detect very small differences in outcomes
Cinical data study: The sample size for the clinical data study was based on an absolute
10% difference in statin prescription rates between intervention and control patients,
a threshold similar to the median effect size found in a systematic review of printed
educational materials; with 80% power and an α-error of 0.05, a sample size of 796 per
group with 20 patients per practice was required
Participants Participating providers:

Administrative data study: all family practices in Ontario; Intervention: 2008 practices;
Control: 1999 practices, number of healthcare professionals not reported;
Cllincal data study: Intervention: 40 practices; Control: 40 practices
Practice type
Administrative data study: Solo: Intervention: 1125 (56.0); Control: 1155 (57.8); Group:
Intervention: 883 (44.0); Control: 844 (42.2)
Rural practice: Intervention:190 (9.5); Control:160 (8.0)
Diabetes patient volume:
< 100; Intervention: 760 (37.8), Control: 708 (35.4)
100 to < 200: Intervention: 742 (37.0), Control: 788 (39.4)
200+ : Intervention: 506 (25.2), Control: 503 (25.2)
Clinical data study:: Solo: Intervention: 16 (40.0), Control: 22 (55.0); Group: Interven-
tion: 24 (60.0), Control: 18 (45.0)
Rural practice ; Intervention: 2 (5.0), Control: 1 (2.5)
Diabetes patient volume
< 100; Intervention:7 (17.5), Control: 4 (10.0)
100 to 200: Intervention: 23 (57.5), Control: 15 (37.5)
200+ : Intervention:10 (25.0), Control: 21 (52.5)
Shah 2014 (Continued)
Patients:
Administrative data study:all diabetic patients > 40 years of age in Ontario; Intervention:
n = 467,713; Control: 466,076
Clinical data study:n of participating patients: Intervention: n = 795; Control: 797 pa-
tients with diabetes aged > 18 years who were seen in the office at least once between
July 2009 and March 2010, and who fulfil the Clinical Practice Guidelines’ definition
of being at ”high risk for CV events“:
Exclusion criteria: Residents of long-term care facilities. Individuals who could not be
assigned to a family practice were excluded
Characteristics of participants:
Administrative data study:
Age, mean (SD): Intervention: 64.3 (12.4); Control: 64.2 (12.4)
Male: Intervention: 246,741 (52.8); Control: 245,204 (52.6)
Diabetes type: no information
Diabetes duration, yrs: < 2 Intervention: 76,547 (16.4), Control: 77,011 (16.5)
yrs 2 to < 5: Intervention:112,509 (24.1), Control: 112,543 (24.1)
yrs 5 to < 10: Intervention: 127,375 (27.2), Control: 126,831 (27.2)
yrs 10+: Intervention: 151,282 (37.3), Control: 149,691 (32.1)
Previous cardiovascular disease: Intervention: 30,108 (6.4), Control: 29,801 (6.4)
Hypertension; Intervention:318,015 (68.0), Control: 317,941 (68.2)
Clinical data study:
Age, mean (SD), y Intervention:65.9 (10.3), Control: 65.5 (10.6)
Male: Intervention: 412 (51.8), Control: 429 (53.8)
Diabetes type: Type 1 14 (1.8) 11 (1.4); Type 2 781 (98.2) 786 (98.6)
Diabetes duration, y: <2 Intervention: 145 (18.2), Control: 120 (15.1)
2-5 Intervention:196 (24.7), Control: 183 (23.0)
5-10 Intervention: 195 (24.5), Control: 214 (26.9)
10+ Intervention: 252 (31.7), Control: 275 (34.5)
Previous cardiovascular disease Intervention: 317 (39.9), Control: 331 (41.5)
Hypertension: Intervention:754 (94.8), Control: 767 (96.2)
Losses to follow-up and withdrawals: No information about the clinical study (none
from the administrative study)
Setting: Family practices
Location (rural/urban):urban and rural Ontario areas
Country: Canada
Interventions Aims: To evaluate the effectiveness of an educational toolkit focusing on cardiovascular

disease screening and risk reduction in people with diabetes
Type of intervention: Passive
Type of guideline tool: printed educational materials (CVD toolkit):The cardiovascular
disease toolkit was a collection of printed educational materials, packaged in a brightly-
coloured box with CDA branding, sent to Canadian family physicians. The contents
included an introductory letter from the Chair of the practice guidelines’ Dissemina-
tion and Implementation Committee; an eight-page summary of selected sections of
the practice guidelines targeted towards family physicians; a four-page synopsis of the
key guideline elements pertaining to cardiovascular disease risk; a small double-sided
laminated card with a simplified algorithm for cardiovascular risk assessment, vascular
protection strategies, and screening for cardiovascular disease; and a pad of tear-off sheets
for patients with a cardiovascular risk self-assessment tool and a list of recommended
risk reduction strategies

Guideline developers: Canadian Diabetes Association (CDA)
Delivery: By mail
Timing: Delivered simultaneously with the updated guideline
Duration of intervention: One-off
Control: Control providers received the Canadian Diabetes Association newsletter,
which included the revised GL
Follow-up time: 10 months
Outcomes Main outcomes:

• Death or non-fatal MI (administrative data study)
• Prescription for statin (clinical data study)
Other (secondary) outcomes:
• Clinical events (admin study): all-cause death, MI, MI or unstable angina, stroke,
stroke or TIA, and other composite outcomes
• CAD assessment (admin study): electrocardiogram, cardiac stress test and nuclear
imaging, coronary angiography, coronary revascularisation processes, cardiology or
internal medicine visits
• Medication initiation (admin study): ACEI/ARB, statin, glucose-lowering drug,
insulin, and nitrate
• Proportion of patients prescribed an angiotensin converting enzyme inhibitor or
angiotensin blocker (clinical study)
• Various intermediate measures (e.g. HbA1c, BP etc) (clinical study)
• Clinical inertia.(clinical study)
Notes Ethical approval and informed consent obtained (yes/no): The study was approved
by the Research Ethics Board of Sunnybrook Health Sciences Centre, Toronto, Ontario
Conflict of interest: BRS was a member of the Guideline Dissemination and Imple-
mentation Committee and the National Research Council of the Canadian Diabetes
Association (CDA) at the time of the study. OB was a member of the Executive of the
Clinical and Scientific Section and the Guideline Dissemination and Implementation
Committee of the CDA at the time of the study. CHYY is currently Chair of the Guide-
line Dissemination and Implementation Committee of the CDA. MMM has served as
an Advisory Board member for the following pharmaceutical companies: Astra Zeneca,
Bristol-Myers Squibb, Eli Lilly and Company, Glaxo Smith Kline, Hoffman La Roche,
Novartis, Novo Nordisk, and Pfizer. JAP has served as both a guest academic editor and
a reviewer for PLOS Medicine
Funding: The study was funded by an operating grant from the Canadian Institutes for
Health Research (CIHR) and the Heart and Stroke Foundation of Canada. BRS receives
salary support from the CIHR, and previously received support from the Canadian
Diabetes Association. The Institute for Clinical Evaluative Sciences (ICES) is a non-
profit research institute funded by the Ontario Ministry of Health and Long-Term Care
(MOHLTC)
Risk of bias
Random sequence generation (selection Low risk “..family practices in the province of On-
bias) tario were allocated 1:1 into the interven-
tion or control group using random num-
ber sequences generated by SAS version 9.
3 (SAS Institute Inc.), stratified by the 14
health regions into which responsibility for
health care delivery in Ontario is divided.
We randomly selected practices from each
of the intervention and control arms, and
randomly selected one physician from each
practice. Each selected physician was con-
tacted, and if willing to participate in the
study,we randomly selected 20 diabetic pa-
tients who had visited the physician dur-
ing the study period, and who fulfilled the
CDA’s definition of being at “high risk for
cardiovascular eventsPatients were selected
using random number sequences generated
by SAS version 9.3 (SAS Institute). Their
charts were reviewed by a trained and expe-
rienced registered nurse, blinded to treat-
ment allocation, who abstracted relevant
data into a computerized data collection
template”
Allocation concealment (selection bias) Low risk See above
Blinding of participants and personnel Low risk In the clinical study the family physicians
(performance bias) were aware they were part of an interven-
All outcomes tion trial, but data were retrospectively re-
trieved
Blinding of participants and personnel Low risk Patients did not know they were part of a
(non-objective outcomes) trial, and data were retrospectively retrieved
Blinding of outcome assessment (detection Low risk Objective outcomes and data (prescription
bias) of statins) retrieved from patient records
All outcomes
Baseline characteristics similar High risk Patient characteristics were similar in the
clinical study. However, there were impor-
tant differences in the type of practices be-
tween groups: more then double the pro-
portion of clinics with 200+ diabetes pa-
tients, and greater proportion of solo prac-
tices in the control group, compared to the
intervention group
Baseline outcome measures similar Low risk No baseline measure of outcomes in the
clinical data study, but baseline levels of
statins prescribed reported in the admin
study
Incomplete outcome data (attrition bias) Low risk The clinical study data were collected from
All outcomes patient records
Selective reporting (reporting bias) Unclear risk Some of the outcomes listed in the trial pro-
tocol (i.e. BMI and waist circumference)
were not reported in the paper
Other bias Unclear risk Endocrinologist in Ontario also received

the intervention tool, but were not part of
the study. However, 18% of the diabetes
patients in Ontario were treated by both
GPs and endocrinologists, which may have
biased the results
ACEI/ARB: angiotensin converting enzyme inhibitor/angiotensin receptor blocker

BMI: body mass index
BP: blood pressure
CCECQA: Committee for Co-ordinating Clinical Evaluation and Quality in Aquitaine
C-RCT: cluster-randomised controlled trial
HbA1c: glycosylated haemoglobin
ICC: intraclass correlation coefficient
MI: myocardial infarction
PORT: Pneumonia Patient Outcomes Research Team
SD: standard deviation
TIA: transient ischaemic attack
Characteristics of excluded studies [ordered by study ID]
Study Reason for exclusion
Allegranzi 2013 Evaluation of WHO hand-hygiene guideline kit. Ineligible study design
Baker 2001 The guideline implementation tools were not developed by existing guideline producers
Bosch 2014 Protocol of a cluster-RCT. Control will not receive guideline only
Chan 2013 One of the authors (not an existing guideline developing body) developed both the guideline and the tool.
Comparison was not guideline only
(Continued)
De Beurs 2015 Eligible intervention and study design, but ineligible outcomes
Eccles 2002 C-RCT: Tools not developed by existing guideline developers
Eccles 2007 C-RCT.Tools not developed by existing guideline developers
Flottorp 2002 Tools not developed by existing guideline developers
Fretheim 2006 Tools not developed by existing guideline developers
Overbeek 2010 Tool not developed by existing guideline developers
Robling 2002 The tool was not developed by existing guideline producers
Rood 2005 Tools not developed by existing guidelines developers
Rycroft-Malone 2012 Tools not developed, but supported by, the guideline developers (RCN/RCA)
Witt 2004 Tool not developed by existing guideline developers
Characteristics of studies awaiting assessment [ordered by study ID]
Maximov 2012
Methods Study design: C-RCT
Participants Healthcare providers: 16 general practitioners/clusters who completed the trial (10 from the intervention group, 6
from the usual-care group)
Patients: 92 patients with knee and hip osteoarthritis (63 in the intervention group, 29 in the usual-care group)
Interventions 1-day didactic educational meeting, provision of the printed guideline and patient brochures
Outcomes Patient’s outcomes investigated: WOMAC pain and stiffness scores, body mass index and self-reported treatment
received (oral NSAID, physical exercise, alternative treatment) at 6 and 12 months after the intervention
Notes Conference abstract only
Van Driel 2007
Participants Healthcare providers: general practitioners in Flanders, Belgium
Interventions Quality circles: self-led meetings using material introduced to the group moderator by a member of the research team
Van Driel 2007 (Continued)
Outcomes Adherence to guidelines
Notes
Characteristics of ongoing studies [ordered by study ID]
Salbach 2014
Trial name or title
Methods C-RCT
Participants Healthcare providers: 20 rehabilitation hospitals/inpatient stroke rehabilitation teams

Patients: people suffering from stroke
Interventions Multicomponent intervention: 2 clinician facilitators per hospital attended a 2-day workshop to receive train-
ing to apply a treatment guideline (18 recommendations) and identify barriers and strategies for implemen-
tation. They also received copies of the treatment recommendations, treatment protocols, presentation slides,
pocket cards, and protected time weekly to facilitate implementation over a 10-week period
Control condition: Copies of the treatment recommendations (the guideline), a video, and a handbook on
using outcome measures
Outcomes Rate of implementation of guideline recommendations
Starting date
Contact information
Notes Conference abstract only.
Te Boveldt 2011
Trial name or title
Participants Healthcare providers: 6 oncology outpatient clinics of hospitals in the South-eastern region of the Netherlands,
with 3 hospitals in the intervention and 3 in the control condition
Interventions A Short Message Service with Interactive Voice Response (SVSIVR) will be used with the aim to improve
pain reporting, pain measurement and adequate pain therapy for people with cancer. The intervention also
includes training of professionals (medical oncologists, nurses, and general practitioners)
Outcomes Pain reporting, pain measurement, adequate pain therapy and pain intensity
Starting date
Te Boveldt 2011 (Continued)
Contact information E-mail: [email protected]

Adress: Department Anesthesiology, Pain and Palliative Medicine, Radboud University Nijmegen Medical
Centre (RUNMC), Nijmegen, 6500 HB, The Netherlands
Notes Protocol only
DATA AND ANALYSES
This review has no analyses.
ADDITIONAL TABLES
Table 1. Guideline development process
Author Year Guideline Literature Critical Consensus pro- Key Barriers/fa-

Targeted developers review appraisal cesses stakeholder in- cilitator assess-
behaviour volvement ment
Bekkering 2005 The Royal CPGs2 Not men- Based on scien- The CPGs were Barriers to
Targeted Dutch Physio- were constructed tioned but prob- tific evidence. pi- change were as-
behaviour: therapy Associa- on the basis of ably included in If no ev- lot-tested among sessed through a
management of tion. the phases of the the Dutch idence was avail- 100 physiother- survey as part of
non-specific physio- method of devel- able, consensus apists and re- the CPG devel-
low back pain therapy process, oping CPGs between experts viewed by an ex- opment process
Number of rec- using the Dutch was obtained ternal multidis-
ommendations: method of ciplinary panel
4 main recom- developing phys-
mendations iotherapy guide-
lines, and evi-
dence from sys-
tematic reviews
were identified
through search-
ing electronic
databases
Daucourt 2003 The Committee The CPG devel- - CPG3 devel- - -

Targeted be- for Co-ordinat- opers opment involved
haviour: appro- ing conducted an expert con-
priate thyroid Clinical Evalua- a comprehensive sensus process.
function testing tion and Quality review of the lit-
Number of rec- in erature
ommendations: Aquitaine
not reported (CCECQA) de-
veloped
guidelines in col-
laboration with
a regional work-
ing group and
a national review
group
Fine 2003 Researchers who The CPG was - The CPG devel- The -
Targeted be- were part of the based on a re- opment guideline was re-
haviour: appro- Pneumo- view of the med- process involved viewed by clini-
priate duration nia Patient Out- ical literature, the consensus of cal opinion lead-
of intravenous comes Research and empiric evi- an 8-member na- ers at each study
Table 1. Guideline development process (Continued)
antibiotic ther- Team (PORT) dence on time to tional guideline site, and was ap-
apy for treat- reach clinical sta- panel proved for local
ment of pneu- bility use by the rel-
monia evant utilisation
Number of rec- management de-
ommendations: partment
a 2-step recom-
mendation
Shah 2014 Canadian Expert Commit- After formulat- Based on scien- A draft docu- -
Targeted be- Diabetes Associ- tee members ing tific evidence/re- ment was circu-
haviour: man- ation (and Ex- evaluated the rel- new recommen- view of the liter- lated nationally
agement of car- pert Committee evant literature, dations or modi- ature and
diovascular risk members) and guidelines fying existing internation-
factors and out- were developed ones based ally for review by
comes of car- and initially re- on new evidence, numerous stake-
diovascular dis- viewed each recommen- holders and
ease in people by the Expert dation was experts in rele-
with Committee assigned a grade vant fields.
diabetes from A through Subse-
Number of rec- D quently, a panel
ommendations: of 6 methodolo-
no information gists,
who were not
directly involved
with the initial
review and
assessment of
the evidence, in-
dependently re-
viewed each
recommenda-
tion, its assigned
grade and sup-
portive citations
1 Khunti 1998. Development of evidence-based review criteria for the management of patients with depression in general practice. No
published version of the guideline found.
2 Bekkering 2003. Dutch physiotherapy guidelines for low back pain.
3
Saillour Glénisson 2001. Guidelines for thyroid function tests in adults.
4 Shah 2014. Canadian Diabetes Association Clinical Practice Guidelines Expert Committee: Canadian Diabetes Association 2008
clinical practice guidelines for the prevention and management of diabetes in Canada.
Table 2. Guideline tool development and delivery
Author Year Delivery of the in- Theoreti- Evidence base Targeted to barri- Key stakeholder
tervention cal models/ frame- ers involvement
works used
Bekkering 2005 Mode: face-to-face - “The sessions were The content of the 2 experts gave ad-
Type of interven- Provider: The pri- based on interven- strategy was deter- vice on the content
tion: mary investiga- tions that have all mined on the ba- of the strategy
simple, but multi- tor and 1 of 2 addi- been shown to be ef- sis of information
component; active tional trainers with fective, such as in- about the expected
only adequate clinical ex- teractive ed- barriers for imple-
Intervention tar- perience in the man- ucation and discus- mentation gathered
get: the healthcare agement of low back sion, feedback, and during the develop-
professional pain supervised the reminders”.1,2,3,4,5 ment of the CPGs
Targeted training sessions
behaviour:
management
of nonspecific low
back pain
Daucourt 2003 Mode: Paper-based - “Among the clinical - -

Type of interven- materials guide-
tion: Provider: none line diffusion strate-
multicomponent; gies, the most effec-
passive only tive are feedback, re-
Intervention tar- minders, academic
get: the healthcare detailing and finan-
professional cial incentives1,2,4,5
Targeted be- Administrative pro-
haviour: appropri- cedures such as the
ate thyroid func- implementation of
tion testing test request forms
have also proved ef-
fective.”
Fine 2003 Mode: Paper-based - “The multifaceted - -

Type of interven- material guideline dissemi-
tion: (detail sheets/treat- nation strategy con-
single; active + pas- ment recommenda- sisted of interven-
sive tions in tions of proven ben-
Intervention tar- patient records) and efit, including real-
get: the healthcare telephone reminder time physician re-
professional Provider: nurse de- minders” 1,6,7,8
Targeted livered telephone re-
behaviour: appro- minder
priate duration of
intravenous antibi-
otic therapy for
treatment of pneu-
monia
Table 2. Guideline tool development and delivery (Continued)
Shah 2014 Mode: printed edu- The implicit theory “The lit- - The toolkit was
Type of interven- cational materials behind its erature has demon- created for the CDA
tion: Targeting the fam- development strated that the ben- by clinical experts
passive ily physician: was that the guide- efits of printed ed- including family
Intervention The cardiovascular lines were too long ucational interven- physicians, endocri-
target: disease toolkit was a and complex to be tions are, at best, nologists, and other
fam- collection of printed easily incorporated modest. A system- healthcare
ily physicians (and educational materi- into clinical prac- atic review of meth- professionals,
diabetes patients at als, pack- tice, so the toolkit ods to improve prac- with guidance from
high risk of cardio- aged in a brightly- aimed to simplify tice guideline adher- clinicians with ex-
vascular disease) coloured box with the information, tai- ence demon- pertise in knowl-
Targeted be- CDA branding, sent lor it towards clini- strated an absolute edge
haviour: manage- to Canadian fam- cal practice, and improvement of 8% translation and im-
ment of cardiovas- ily physicians. The provide explicit ac- for educational ma- plementation
cular risk factors contents in- tionable recommen- terials. A more re-
and outcomes of cluded an introduc- dations cent Cochrane re-
cardiovascular dis- tory letter from the view found
ease in people with Chair of the practice that printed educa-
diabetes guidelines’ Dis- tional materials led
semination and Im- to a median ab-
plementation Com- solute improvement
mittee; an in performance of
8-page summary of only 2% (25). Stud-
selected sections of ies of printed mate-
the practice guide- rials specifically tied
lines to clin-
targeted to- ical practice guide-
wards family physi- lines also showed
cians; a 4-page syn- modest benefits. A
opsis of the key small English study
guideline elements randomised 42 fam-
pertaining to car- ily physicians to re-
diovascular disease ceive an algorithm
risk; a small dou- for monitoring and
ble-sided laminated treatment of hyper-
card with a simpli- tension of diabetic
fied algorithm for patients based on
cardiovascu- practice guidelines,
lar risk assessment, but found no differ-
vascular protection ence in
strategies, and blood pressure con-
screening for car- trol between the in-
diovascular disease, tervention and con-
and a pad of tear- trol groups. How-
off sheets for pa- ever, some processes
tients with a cardio- of care were slightly
vascular risk self-as-
Table 2. Guideline tool development and delivery (Continued)
sessment tool and a improved:

list patients in the inter-
of recom- vention group were
mended risk reduc- prescribed higher
tion strategies doses of antihyper-
Provider: NA13 tensive medications,
and had more physi-
cian visits to mon-
itor blood pressure.
In a larger Cana-
dian study, family
physicians were ran-
domised to receive
a 1-page summary
of a 3-year-old prac-
tice guideline on
anti-anginal therapy
from the local med-
ical governing body.
No differences were
noted in prescrip-
tion of β-blockers,
an-
tiplatelet agents, or
lipid-lowering drugs
between groups in
the 7000 patients re-
viewed” 9,10,11,12
1 Bero 1998 Closing the gap between research and practice: an overview of systematic reviews of interventions to promote the imple-
mentation of research findings.
2 Davis 1995 Changing physician performance. A systematic review of the effect of continuing medical education strategies.
3 Wensing 1998 Implementing guidelines and innovations in general practice: which interventions are effective?
4
Grimshaw 1995 Developing and implementing clinical practice guidelines.
5 Davis 1997. Translating guidelines into practice. A systematic review of theoretic concepts, practical experience and research evidence
in the adoption of clinical practice guidelines.

6 Murrey 1992 Implementing clinical guidelines: a quality management approach to reminder systems.
7 Grimshaw 1993. Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations.
8 Weingarten 2000. Translating practice guidelines into patient care: guidelines at the bedside.
9
Grimshaw 2006. Towards evidence-based quality improvement: evidence (and its limitation) of the effectiveness of guideline dissem-
ination and implementation strategies 1966-1998.
10 Giguère 2012. Printed educational materials: effects on professional practice and healthcare outcomes.
11 Bebb 2007. A cluster randomised controlled trial of the effect of a treatment algorithm for hypertension in patients with type 2
diabetes.
12 Beaulieu 2004. Drug treatment of stable angina pectoris and mass dissemination of therapeutic guidelines: a randomized controlled
trial.
13 Not applicable
Table 3. Intervention components
Author Year Tailoring Feedback Educational out- Reminders (pa- Decision support Other
reach/ per, electronic, tools (test order forms,
Academic detail- telephone) supportive mate-
ing/ Small group rials etc.)
discussions
Bekkering 2005 The content of - 2 interactive train- - - -

the strategy was ing sessions, each
determined lasting 2½ hours,
on the basis of infor groups of 8
formation about - 12 physiothera-
the expected bar- pists (in-
riers for imple- cluding feedback
mentation gath- on current man-
ered dur- agement and re-
ing the develop- minders). For each
ment of the clin- session a prepa-
ical guidelines ration time of 2
hours was recom-
mended
Daucourt 2003 - - - Pocket memoran- - Test request form.

dum card.
Fine 2003 - - - Paper- - -

based detail sheet/
treatment recom-
mendations put
into the patient’s
record + real-time
nurse telephone
reminder
Shah 2014 - - - - - Printed educa-

tional materials
Table 4. Results: Other outcomes
Author Year Clinical outcomes; Quality of life Mortality Health- Costs

Medical complica- Satisfaction with care resource use
tions care (including medica-
tions prescribed)
Bekkering 2005 - Quality of Life - - Mean annual cost

(assessed with the EQ- (Euros) per patient
5D1 ), mean (SD): (SD):
BL: Direct cost2 :
Inter- Intervention: 374
Table 4. Results: Other outcomes (Continued)
vention: 0.6730 (0. (437).

2042); Control: 0. Control: 449 (572).
6134 (0.2661), The costs (Euro) of re-
P = 0.006. leasing a new guide-
At 6 weeks: line for low back pain
Inter- to 18,000
vention: 0.7778 (0. physiotherapists:
1978); Control: 0. Intervention (active
7497 (0.2316) strategy): 87,416
At 12 weeks: Control (passive
Inter- strategy): 63,101
vention: 0.8141 (0.
1988); Control: 0.
7873 (0.2210)
Note: results for 26
and 52 weeks re-
ported graphically
Daucourt 2003 - - - - Cost paper awaits

translation
Fine 2003 In-hospi- Patient satisfaction Mortality6 Length of index hos- -

tal medical complica- with care4 , number all-cause, pital (days) stay, me-
tions, number (%): (%): number (%): Inter- dian (IQR): Inter-
In- Not satisfied with vention: 22 (8), n = vention: 5.0 (3.0 to
tervention:157 (55); overall care: Inter- 283; Control: 29 (9) 7.0); Control: 5.0 (3.
Control: 206 (63), P vention: 12 (5.3), n , n = 325, P = 0.70 0 to 8.0); Hazard ra-
= 0.04 = 228; Control: 11 Pneumonia-re- tio (95% CI): 1.16
Functional status3 (4.0), n = 273, P = 0. lated mortality, num- (0.97 to 1.38), P = 0.
SF-12 phys- 67 ber (%): Interven- 11
ical health compos- Be- tion: 15 (5); Con- Rehospitali-
ite score: Interven- lieved length of stay trol: 23 (7), P = 0.44 sation7 number (%):
tion:45 ± 7, n = 181; was too short: Inter- Intervention:37 (14)
Control: 45 ± 7, n = vention: 59 (26.1); ; Control:33 (11), P
223; P = 0.71 Control: 54 (20.2), = 0.42
SF-12 mental health P = 0.16 Duration (days)
composite score: In- Return to usual ac- of intravenous antibi-
tervention: 45 ± 6; tivities5 ,Hazard ratio otic therapy, median
Control: 45 ± 7, P = (95% CI): (IQR): Intervention:
0.71 Nonworkers: 1. 3.0 (2.0 to 5.0),n =
09 (0.83 to 1.43), P 283; Control: 4.0 (2.
= 0.55 6 to 6.0), n = 325;
Workers: 0.85 (0.54 Hazard ratio (95%
to 1.35); P = 0.49 CI): 1.23 (1.00 to 1.
Return to work 52), P = 0.06
(workers) 0.99 (0.63
to 1.58),
P = 0.98
Shah 2014 Clinical data study: - Administrative Clinical data study: -

Intervention: n = data study : Inter- Primary outcome:
40 practices/795 pa- vention: 2008 prac- Proportion of par-
tients; Control: n = tices (467,713 participants prescribed
40 practices/797 pa- ticipants); Control: statins (new or re-
tients 1999 practices (466, newed prescription)
Cardiovascu- 076 participants) : Intervention: 700
lar risk reduction (sec- Primary outcome: (88.1%); Control:
ondary outcomes): Death or non-fa- 725 (90.1%); OR 0.
Proportion of partic- tal myocardial in- 73, 95% CI 0.42 to
ipants reaching gly- farction: Interven- 1.26, P = 0.26
caemic control tar- tion: 2.5%; Control: Proportion of partic-
gets (HbA1c < 7. 2.5%; OR 1.00 (0. ipants prescribed an
0%): Intervention: 96 to 1.03), P = 0.77 ACEI/ARB: In-
58.5%; Control: 58. Secondary tervention: Control:
8%; OR 0.93 (0.71 outcomes: Medica- Secondary outcome.
to 1.21), P = 0.58 tion ini- Administrative
Proportion of partic- tiation (ACEI/ARB data study :
ipants reach- > 1 antihypertensive Secondary out-
ing blood pressure class, or > 2, or > 3, comes: CAD assess-
control targets (< statin, glucose-low- ment (electrocardio-
130/80): Interven- ering drugs, insulin, gram, cardiac stress
tion: 52.8%; Con- nitrate): OR, range: test, nuclear imag-
trol: 63.5%, OR 0. from 0.96 to 1.02, P ing, coronary an-
72 (0.53 to 0.98), P values from 0.03 to giography, coronary
= 0.04 0.94 revascularisation,
Proportion of partic- cardiology or inter-
ipants reach- nal medicine visit):
ing LDL-cholesterol OR, range: from 0.
control targets (< 2.0 96 to 1.00, P values
mmol/L): Interven- from 0.02 to 0.83
tion: 59.2%; Con-
trol: 61.7% , OR 0.
90 (0.68 to 1.18), P
= 0.43
Proportion of partic-
ipants reaching To-
tal to HDL-choles-
terol ratio (< 4.0)
: Intervention: 74.
2%; Control: 76.
8%, OR 0.85 (0.63
to1.14), P = 0.27
Clinical (secondary
outcomes):
When HbA1c >
8.0%: Intervention:
11.8%; Control: 13.

0%, OR 0.98 (0.48
to 1.98), P = 0.95
When blood pres-
sure > 140/90: Inter-
vention: 5.6%; Con-
trol: 7.2%, OR 0.67
(0.25 to 1.82), P = 0.
43
When LDL choles-
terol > 3.0 mmol/
L: Intervention: 43.
5%; Control: 45.
2%, OR 0.94 (0.53
to 1.67), P = 0.83
Administrative
data study : Inter-
vention: 2008 prac-
tices (467,713 par-
ticipants); Control:
1999 practices (466,
076 participants)
Secondary
outcomes:
Clin-
ical events (all-cause
death, MI, MI or un-
stable angina, stroke,
stroke or TIA, and
combined out-
comes): OR: from 0.
98 to 1.04, P values
from 0.21 to 0.96
1 EQ-5D: a standardised instrument for use as a measure of health outcome.The EQ-5D has five dimensions: mobility, self-care, usual
activity,pain/discomfort and anxiety/depression. Each dimension has three levels, no problems, some problems and serious problems.
Hence, EQ-5D has 243 possible health states. Utility values of the general public for these health states as measured with the time
tradeoff technique on a random sample of the adult population of the United Kingdom, the MVH-A1 tariff, were applied in this
study. The scores range from −0.594 (worst situation) to 1.0 (perfect health).
2 The direct costs consisted of costs of the dissemination of the guideline and the costs of the healthcare utilisation of the patients. Prices
for the year 2002.

3 SF-12 health scores were assessed in all patients able to provide reliable self-report data during the 30-day interview, excluding 6
intervention-arm and 6 control-arm patients with missing data.

4 Patient satisfaction with care was assessed for all patients with a 30-day interview that was not completed by a paid caregiver, excluding
four intervention-arm and two control-arm patients with missing data. An additional two patients in the intervention arm and six
patients in the control arm were hospitalised for the full 30 days and were not asked about length of hospital stay. SF-12, 12-Item Short
Form was used.
5 Return to usual activities among non-workers was assessed for 183 intervention arm and 219 control arm patients not employed
at baseline who completed a 30-day interview. Return to usual activities among workers was assessed in 59 intervention-arm and
59 control-arm patients employed at baseline. Return to work was assessed among 54 intervention-arm and 53 control-arm patients
employed at baseline.
6 Mortality, medical complications, and return to work and usual activities were adjusted for pneumonia severity risk class.
7 Rehospitalisation within 30 days of the index admission was assessed for all patients who were discharged alive from either the index
hospitalisation or another acute-care facility (if transferred to an acute-care facility from the index hospitalisation).
8 Fluid fasting times assessed by local investigator asking the patient about the fasting time, and checking this information against
medical notes.
9 Cost for designing, editing, reproducing, and posting need when applied to 170 acute trusts.
10 Cost of providing 170 acute trusts with implementation support through a web-based resource championed through opinion
leadership. This includes development costs for the tool (which for this project were in-house costs, in other cases external agencies may
have to be used which are likely to be three times higher), publicity materials, training materials and opinion leader time and activity.
Table 5. Results: Adherence outcomes
Author Year Adherence Participants (Set- Control Adherence Intervention Median ARD
Outcomes tings) Adherence
Bekkering 2005 Adher- 113 i) Post: i) Post: +0.115

(Hoijenboos 2005) ence to 4 guideline physiotherapists 14 (13), n = 253 32 (27), n = 247 11.5% higher ad-
Targeted be- recommendations: (68 private physio- ii) Post: ii) Post: 188 (79) herence in the inter-
haviour: manage- i) Limit number of therapy practices) 180 (71) iii) Post: 183 (77) vention group
ment of non spe- sessions in normal iii) Post: iv) Post: 229 (96) i) 0.14%
cific low back pain course back pain 154 (60) ii) 0.08%
GL tool used: ii) Set functional iv) Post: iii) 0.17%
interactive training treatment goals 221 (87) iv) 0.09%
workshop X2 iii) Use mainly ac-
tive interventions
iv) Give adequate
information
Note: an increase
was desirable for all
outcomes
Daucourt 2003 Global Guideline 1412 physicians Pre: 62.0% Dual inter- +0.159%
Targeted be- Conformity Rate (6 general hospitals) (95% CI 47.7 to 76. vention group: Post:
haviour: appropri- 4) 77.9% (95% CI 68.
ate thyroid func- 9 to 87.0)
tion testing Note: only results for
Guideline tool the dual interven-
used: tion presented here
1) Dual interven-
tion (2 + 3);
2) Order request
form;
3) Pocket memo-
randum card
Fine 2003 No 545 physicians - - -

Targeted adherence outcomes (7 not-for-profit
behaviour: appro- reported, only prox- hospitals)
priate duration of ies
Table 5. Results: Adherence outcomes (Continued)
intravenous antibi-
otic therapy for
treatment of pneu-
monia
GL tool used: de-
tail sheet/ treat-
ment recommen-
dations+ telephone
reminder
Shah 2014 No 2 separate studies: - - -

Targeted be- adherence outcomes Administrative data
haviour: improved reported, only prox- study: n = 4007
cardiovascular risk ies practices: Interven-
factor man- tion: 2008; Control:
agement in people 1999
with diabetes Clinical data study.
Guideline tool n = 80 practices
used: printed edu- (1592 patients); In-
cational material tervention: 40 prac-
tices (8795 patients)
; Control: 40 prac-
tices (8797 patients)
APPENDICES
Appendix 1. Medline search strategies

MEDLINE (OvidSP) (1946 to present, In process) - February 2016
# Searches
1 (“United States Agency for Healthcare Research and Quality”/ or Health Maintenance Organization/) and practice guidelines
as topic/
2 (“United States Agency for Healthcare Research and Quality”/ or Health Maintenance Organization/) and Guideline Adherence/
3 (health maintenance organi?ation* or hmo? or Aetna or Blue Cross Blue Shield Association or CIGNA or Kaiser Permanente
or Humana or Health Net or UnitedHealth Group or Wellpoint or AHCPR).ti,in. and (guideline* or guidance or standard*1
or pathway*1 or protocol*1).ti
4 1 or 2 or 3
(Continued)
5 “comment on”.cm. or systematic review.ti. or literature review.ti. or editorial.pt. or meta-analysis.pt. or news.pt. or review.pt
6 4 not 5
7 exp animals/ not humans/
8 6 not 7
9 limit 8 to yr=“1998 -Current”
10 *Guideline Adherence/
11 (guideline* and (implement* or uptake* or adopt* or adhere* or concord* or complian* or comply or non-adhere* or nonadhere*
or non-concord* or nonconcord* or non-complian* or noncomplian*)).ti
12 (guidance and (implement* or uptake* or adopt* or adhere* or concord* or complian* or comply or non-adhere* or nonadhere*
13 (standard? and (implement* or uptake* or adopt* or adhere* or concord* or complian* or comply or non-adhere* or nonadhere*
14 (pathway? and (implement* or uptake* or adopt* or adhere* or concord* or complian* or comply or non-adhere* or nonadhere*
15 (protocol? and (implement* or uptake* or adopt* or adhere* or concord* or complian* or comply or non-adhere* or nonadhere*
16 10 or 11 or 12 or 13 or 14 or 15
17 (national adj3 (guideline*1 or guidance or standard*1 or pathway*1 or protocol*1)).ti,ab
18 (regional adj3 (guideline* or guidance or standard*1 or pathway*1 or protocol*1)).ti,ab
19 (society adj3 (guideline* or guidance or standard*1 or pathway*1 or protocol*1)).ti,ab
20 (association adj3 (guideline* or guidance or standard*1 or pathway*1 or protocol*1)).ti,ab
21 (academy adj3 (guideline* or guidance or standard*1 or pathway*1 or protocol*1)).ti,ab
22 (board adj3 (guideline* or guidance or standard*1 or pathway*1 or protocol*1)).ti,ab
23 (institute? adj3 (guideline* or guidance or standard*1 or pathway*1 or protocol*1)).ti,ab
24 (ministry adj3 (guideline* or guidance or standard*1 or pathway*1 or protocol*1)).ti,ab
25 (department? adj3 (guideline* or guidance or standard*1 or pathway*1 or protocol*1)).ti,ab
(Continued)
26 ((health maintenance organi?ation* or hmo? or Aetna or Blue Cross Blue Shield Association or CIGNA or Kaiser Permanente
or Humana or Health Net or UnitedHealth Group or Wellpoint) and (guideline* or guidance or standard*1 or pathway*1 or
protocol*1)).ti,ab
27 exp Managed Care Programs/ and (guideline* or guidance or standard*1 or pathway*1 or protocol*1).ti,ab,hw
28 17 or 18 or 19 or 20 or 21 or 22 or 23 or 24 or 25 or 26 or 27
29 (implement* or uptake* or adopt* or adhere* or concord* or complian* or comply or non-adhere* or nonadhere* or non-
concord* or nonconcord* or non-complian* or noncomplian*).ti,ab,hw
30 28 and 29
31 practice guidelines as topic/
32 (practice adj3 (guideline*1 or guidance or standard*1 or pathway*1)).ti,ab
33 (clinical adj3 (guideline*1 or guidance or standard*1 or pathway*1 or protocol*1)).ti,ab
34 31 or 32 or 33
35 Guideline Adherence/
36 Health Plan Implementation/
37 (guideline* adj5 (implement* or uptake* or adopt* or adhere* or concord* or complian* or comply or non-adhere* or nonadhere*
or non-concord* or nonconcord* or non-complian* or noncomplian*)).ti,ab
38 (guidance adj5 (implement* or uptake* or adopt* or adhere* or concord* or complian* or comply or non-adhere* or nonadhere*
39 (standard? adj5 (implement* or uptake* or adopt* or adhere* or concord* or complian* or comply or non-adhere* or nonadhere*
40 (pathway? adj5 (implement* or uptake* or adopt* or adhere* or concord* or complian* or comply or non-adhere* or nonadhere*
41 (protocol? adj5 (implement* or uptake* or adopt* or adhere* or concord* or complian* or comply or non-adhere* or nonadhere*
42 35 or 36 or 37 or 38 or 39 or 40 or 41
43 34 and 42
44 16 or 30 or 43
45 “comment on”.cm. or systematic review.ti. or literature review.ti. or editorial.pt. or meta-analysis.pt. or news.pt. or review.pt
(Continued)
46 44 not 45
47 exp animals/ not humans/
48 46 not 47
49 randomized controlled trial.mp. or controlled clinical trial.pt. or randomized.ab. or placebo.ab. or clinical trials as topic.sh. or
randomly.ab. or trial.ti
50 48 and 49
52 Program Evaluation/
53 Program Development/
54 Intervention Studies/
55 intervention*.ti.
56 (intervention* adj6 (clinician* or collaborat* or community or complex or DESIGN* or doctor* or educational or family
doctor* or family physician* or family practitioner* or financial or GP or general practice* or hospital* or impact* or improv*
or individuali?e* or individuali?ing or interdisciplin* or multicomponent or multi-component or multidisciplin* or multi-
disciplin* or multifacet* or multi-facet* or multimodal* or multi-modal* or personali?e* or personali?ing or pharmacies or
pharmacist* or pharmacy or physician* or practitioner* or prescrib* or prescription* or primary care or professional* or provider*
or regulatory or regulatory or tailor* or target* or team* or usual care)).ab
57 (collaborativ* or collaboration* or tailored or personali?ed).ti,ab
58 (exp hospitals/ or exp Hospitalization/ or exp Patients/ or exp Nurses/ or exp Nursing/) and (study.ti. or evaluation studies as
topic/)
59 demonstration project*.ti,ab.
60 (pre-post or “pre test*” or pretest* or posttest* or “post test*” or (pre adj5 post)).ti,ab
61 (pre-workshop or post-workshop or (before adj3 workshop) or (after adj3 workshop)).ti,ab
62 ((study adj3 aim?) or “our study”).ab.
63 (before adj10 (after or during)).ti,ab.
64 (“quasi-experiment*” or quasiexperiment* or “quasi random*” or quasirandom* or “quasi control*” or quasicontrol* or ((quasi*

or experimental) adj3 (method* or study or trial or design*))).ti,ab,hw
65 (“time series” adj2 interrupt*).ti,ab,hw.
(Continued)
66 (time points adj3 (over or multiple or three or four or five or six or seven or eight or nine or ten or eleven or twelve or month*
or hour? or day? or “more than”)).ab
67 pilot.ti.
68 Pilot projects/
69 clinical trial.pt.
70 multicenter study.pt.
71 (multicentre or multicenter or multi-centre or multi-center).ti
72 random*.ti,ab. or controlled.ti.
73 (control adj3 (area or cohort? or compar? or condition or group? or intervention? or participant? or study)).ab
74 (cluster* adj3 (random* or trial*)).ti,ab.
75 52 or 53 or 54 or 55 or 56 or 57 or 58 or 59 or 60 or 61 or 62 or 63 or 64 or 65 or 66 or 67 or 68 or 69 or 70 or 71 or 72 or
73 or 74
76 48 and 75
78 9 or 51 or 77
79 (2013* or 2014* or 2015*).ed,dp,yr.
80 9 and 79
81 51 and 79
82 77 and 79
Appendix 2. Embase search strategy
1 (Health Maintenance Organization/ or managed care organization/ or preferred provider organization/ or pharmacy benefit
manager/) and practice guideline/
2 (Health Maintenance Organization/ or managed care organization/ or preferred provider organization/ or pharmacy benefit
manager/) and (guideline*1 or guidance or standard*1 or pathway*1 or protocol*1).ti
(Continued)
4 1 or 2 or 3
5 review.ti.
6 (animal$ not human$).sh,hw. or (rat or rats or cow or cows or chicken? or horse or horses or mice or mouse or bovine or animal?
).ti
7 5 or 6
8 4 not 7
15 10 or 11 or 12 or 13 or 14
(Continued)
protocol*1)).ti,ab
26 exp managed care/ and (guideline* or guidance or standard*1 or pathway*1 or protocol*1).ti,ab,hw
29 27 and 28
30 *practice guideline/
33 30 or 31 or 32
39 34 or 35 or 36 or 37 or 38
40 33 and 39
41 15 or 29 or 40
(Continued)
42 (random$ or factorial$ or crossover$ or cross over$ or cross-over$ or placebo$ or (doubl$ adj blind$) or (singl$ adj blind$) or
assign$ or allocat$ or volunteer$).ti,ab. or crossover-procedure/ or double-blind procedure/ or randomized controlled trial/ or
single-blind procedure/
43 41 and 42
44 review.ti.
).ti
46 44 or 45
47 43 not 46
49 intervention?.ti. or (intervention? adj6 (clinician? or collaborat$ or community or complex or DESIGN$ or doctor? or educa-
tional or family doctor? or family physician? or family practitioner? or financial or GP or general practice? or hospital? or impact?
or improv$ or individuali?e? or individuali?ing or interdisciplin$ or multicomponent or multi-component or multidisciplin$ or
multi-disciplin$ or multifacet$ or multi-facet$ or multimodal$ or multi-modal$ or personali?e? or personali?ing or pharmacies
or pharmacist? or pharmacy or physician? or practitioner? or prescrib$ or prescription? or primary care or professional$ or
provider? or regulatory or regulatory or tailor$ or target$ or team$ or usual care)).ab
50 (pre-intervention? or preintervention? or “pre intervention?” or post-intervention? or postintervention? or “post intervention?

”).ti,ab
51 (hospital$ or patient?).hw. and (study or studies or care or health$ or practitioner? or provider? or physician? or nurse? or nursing
or doctor?).ti,hw
52 demonstration project?.ti,ab.
53 (pre-post or “pre test$” or pretest$ or posttest$ or “post test$” or (pre adj5 post)).ti,ab
55 trial.ti. or ((study adj3 aim?) or “our study”).ab.
57 (time points adj3 (over or multiple or three or four or five or six or seven or eight or nine or ten or eleven or twelve or month$
58 pilot.ti.
60 random$.ti,ab. or controlled.ti.
(Continued)
61 *experimental design/ or *pilot study/ or quasi experimental study/
62 (“quasi-experiment$” or quasiexperiment$ or “quasi random$” or quasirandom$ or “quasi control$” or quasicontrol$ or (

(quasi$ or experimental) adj3 (method$ or study or trial or design$))).ti,ab
63 (“time series” adj2 interrupt$).ti,ab.
64 or/49-63
65 41 and 64
66 review.ti.
).ti
68 66 or 67
69 65 not 68
70 9 or 48 or 69
71 (2013* or 2014* or 2015*).em,dp,yr.
72 9 and 71
73 48 and 71
74 69 and 71
Appendix 3. Psychinfo search strategy
1 (Health Maintenance Organizations/ or exp Professional organizations/ or Government Agencies/) and Treatment Guidelines/
2 (Health Maintenance Organizations/ or exp Professional organizations/ or Government Agencies/) and (guideline* or guidance
or standard*1 or pathway*1 or protocol*1).ti
(Continued)
4 1 or 2 or 3
5 review.ti.
6 (rat or rats or cow or cows or chicken? or horse or horses or mice or mouse or bovine or animal?).ti. or exp animals/ or animal?.
ti,id,hw
7 5 or 6
8 4 not 7
15 10 or 11 or 12 or 13 or 14
(Continued)
protocol*1)).ti,ab
26 exp Managed Care/ and (guideline* or guidance or standard*1 or pathway*1 or protocol*1).ti,ab,hw
29 27 and 28
30 Treatment Guidelines/
33 30 or 31 or 32
39 34 or 35 or 36 or 37 or 38
40 33 and 39
41 15 or 29 or 40
42 (random or trial* or controlled stud or placebo* or ((singl* or doubl* or trebl* or tripl*) adj2 (blind* or mask*)) or cross over or
crossover or factorial* or assign* or allocat* or volunteer*).ti,ab,hw,id. or treatment effectiveness evaluation/ or mental health
program evaluation/ or exp experimental design/ or “2000”.md
43 41 and 42
(Continued)
44 review.ti.
ti,id,hw
46 44 or 45
47 43 not 46
49 intervention?.ti. or (intervention? adj6 (clinician? or collaborat$ or community or complex or DESIGN$ or doctor? or educa-
tional or family doctor? or family physician? or family practitioner? or financial or GP or general practice? or hospital? or impact?
or improv$ or individuali?e? or individuali?ing or interdisciplin$ or multicomponent or multi-component or multidisciplin$ or
multi-disciplin$ or multifacet$ or multi-facet$ or multimodal$ or multi-modal$ or personali?e? or personali?ing or pharmacies
or pharmacist? or pharmacy or physician? or practitioner? or prescrib$ or prescription? or primary care or professional$ or
provider? or regulatory or regulatory or tailor$ or target$ or team$ or usual care)).ab
50 (pre-intervention? or preintervention? or “pre intervention?” or post-intervention? or postintervention? or “post intervention?

”).ti,ab
51 (hospital$ or patient?).hw. and (study or studies or care or health$ or practitioner? or provider? or physician? or nurse? or nursing
or doctor?).ti,hw
52 demonstration project?.ti,ab.
53 (pre-post or “pre test$” or pretest$ or posttest$ or “post test$” or (pre adj5 post)).ti,ab
55 trial.ti. or ((study adj3 aim?) or “our study”).ab.
57 (“quasi-experiment$” or quasiexperiment$ or “quasi random$” or quasirandom$ or “quasi control$” or quasicontrol$ or (

(quasi$ or experimental) adj3 (method$ or study or trial or design$))).ti,ab,hw
58 (“time series” adj2 interrupt$).ti,ab,hw.
59 (time points adj3 (over or multiple or three or four or five or six or seven or eight or nine or ten or eleven or twelve or month$
60 pilot.ti.
62 random$.ti,ab. or controlled.ti.
(Continued)
63 (control adj3 (area or cohort? or compare? or condition or design or group? or intervention? or participant? or study)).ab. not
(controlled clinical trial or randomized controlled trial).pt
64 experimental design/ or between groups design/ or quantitative methods/ or quasi experimental methods/
65 or/49-64
66 41 and 65
67 review.ti.
ti,id,hw
69 67 or 68
70 66 not 69
72 9 or 48 or 71
73 (2013* or 2014* or 2015*).dp,up,yr.
74 9 and 73
75 48 and 73
76 71 and 73
Appendix 4. Cinahl search strategy
1 ( (MH “Health Maintenance Organizations”) OR (MH “Independent Practice Associations”) OR (MH “Preferred Provider
Organizations”) OR (MH “Provider-Sponsored Organizations”) ) AND (MH “Practice Guidelines”)
2 ( (MH “Health Maintenance Organizations”) OR (MH “Independent Practice Associations”) OR (MH “Preferred Provider
Organizations”) OR (MH “Provider-Sponsored Organizations”) ) AND (MH “Guideline Adherence”)
3 ( TI ( (health maintenance organi?ation* or hmo? or Aetna or Blue Cross Blue Shield Association or CIGNA or Kaiser Permanente
or Humana or Health Net or UnitedHealth Group or Wellpoint or AHCPR) ) OR AF ( (health maintenance organi?ation*
or hmo? or Aetna or Blue Cross Blue Shield Association or CIGNA or Kaiser Permanente or Humana or Health Net or
UnitedHealth Group or Wellpoint or AHCPR) ) ) AND TI ( guideline* or guidance or standard or standards or pathway or
pathways or protocol or protocols )
4 1 OR 2 OR 3
(Continued)
5 TI review
6 (MH “Animals+”) NOT (MH “Human”)
7 TI ( rat or rats or cow or cows or chicken? or horse or horses or mice or mouse or bovine or animal? ) OR MW animal?
8 5 OR 6 OR 7
9 4 NOT 8
10 TI ( (guideline* and (implement* or uptake* or adopt* or adhere* or concord* or complian* or comply or non-adhere* or
nonadhere* or non-concord* or nonconcord* or non-complian* or noncomplian*)) ) OR TI ( (guidance and (implement*
or uptake* or adopt* or adhere* or concord* or complian* or comply or non-adhere* or nonadhere* or non-concord* or
nonconcord* or non-complian* or noncomplian*)) ) OR TI ( (standard? and (implement* or uptake* or adopt* or adhere*
or concord* or complian* or comply or non-adhere* or nonadhere* or non-concord* or nonconcord* or non-complian* or
noncomplian*)) ) OR TI ( (pathway? and (implement* or uptake* or adopt* or adhere* or concord* or complian* or comply
or non-adhere* or nonadhere* or non-concord* or nonconcord* or non-complian* or noncomplian*)) ) OR TI ( (protocol?
and (implement* or uptake* or adopt* or adhere* or concord* or complian* or comply or non-adhere* or nonadhere* or non-
concord* or nonconcord* or non-complian* or noncomplian*)) )
11 TI ( (national n3 (guideline*1 or guidance or standard or standards or pathway or pathways or protocol or protocols)) ) OR

TI ( (regional n3 (guideline* or guidance or standard or standards or pathway or pathways or protocol or protocols)) ) OR TI
( (society n3 (guideline* or guidance or standard or standards or pathway or pathways or protocol or protocols)) ) OR TI (
(association n3 (guideline* or guidance or standard or standards or pathway or pathways or protocol or protocols)) ) OR TI
( (academy n3 (guideline* or guidance or standard or standards or pathway or pathways or protocol or protocols)) ) OR TI
( (board n3 (guideline* or guidance or standard or standards or pathway or pathways or protocol or protocols)) ) OR TI (
(institute? n3 (guideline* or guidance or standard or standards or pathway or pathways or protocol or protocols)) ) OR TI (
(ministry n3 (guideline* or guidance or standard or standards or pathway or pathways or protocol or protocols)) ) OR TI (
(department? n3 (guideline* or guidance or standard or standards or pathway or pathways or protocol or protocols)) ) OR TI ( (
(health maintenance organi?ation* or hmo? or Aetna or Blue Cross Blue Shield Association or CIGNA or Kaiser Permanente or
Humana or Health Net or UnitedHealth Group or Wellpoint) and (guideline* or guidance or standard or standards or pathway
or pathways or protocol or protocols)) )
12 AB ( (national n3 (guideline*1 or guidance or standard or standards or pathway or pathways or protocol or protocols)) ) OR

AB ( (regional n3 (guideline* or guidance or standard or standards or pathway or pathways or protocol or protocols)) ) OR AB
( (society n3 (guideline* or guidance or standard or standards or pathway or pathways or protocol or protocols)) ) OR AB (
(association n3 (guideline* or guidance or standard or standards or pathway or pathways or protocol or protocols)) ) OR AB
( (academy n3 (guideline* or guidance or standard or standards or pathway or pathways or protocol or protocols)) ) OR AB
( (board n3 (guideline* or guidance or standard or standards or pathway or pathways or protocol or protocols)) ) OR AB (
(institute? n3 (guideline* or guidance or standard or standards or pathway or pathways or protocol or protocols)) ) OR AB (
(ministry n3 (guideline* or guidance or standard or standards or pathway or pathways or protocol or protocols)) ) OR AB (
(department? n3 (guideline* or guidance or standard or standards or pathway or pathways or protocol or protocols)) ) OR AB (
((health maintenance organi?ation* or hmo? or Aetna or Blue Cross Blue Shield Association or CIGNA or Kaiser Permanente or
Humana or Health Net or UnitedHealth Group or Wellpoint) and (guideline* or guidance or standard or standards or pathway
or pathways or protocol or protocols)) )
13 (MH “Managed Care Programs+”) AND TI ( guideline*1 or guidance or standard or standards or pathway or pathways or
protocol or protocols )
(Continued)
14 (MH “Managed Care Programs+”) AND AB ( guideline*1 or guidance or standard or standards or pathway or pathways or
15 (MH “Managed Care Programs+”) AND MW ( guideline*1 or guidance or standard or standards or pathway or pathways or
16 11 OR 12 OR 13 OR 14 OR 15
17 TI ( implement* or uptake* or adopt* or adhere* or concord* or complian* or comply or non-adhere* or nonadhere* or

non-concord* or nonconcord* or non-complian* or noncomplian* ) OR AB ( implement* or uptake* or adopt* or adhere*
noncomplian* ) OR MW ( implement* or uptake* or adopt* or adhere* or concord* or complian* or comply or non-adhere*
or nonadhere* or non-concord* or nonconcord* or non-complian* or noncomplian* )
18 16 AND 17
19 (MH “Practice Guidelines”)
20 TI ( (practice n3 (guideline*1 or guidance or standard*1 or pathway*1)) ) OR AB ( (practice n3 (guideline*1 or guidance or

standard*1 or pathway*1)) ) OR TI ( (clinical n3 (guideline*1 or guidance or standard*1 or pathway*1 or protocol*1)) ) OR
AB ( (clinical n3 (guideline*1 or guidance or standard*1 or pathway*1 or protocol*1)) )
21 19 OR 20
22 (MH “Guideline Adherence”)
23 (MH “Program Implementation”) OR (MH “Systems Implementation”)
24 TI ( (guideline* n5 (implement* or uptake* or adopt* or adhere* or concord* or complian* or comply or non-adhere* or

nonadhere* or non-concord* or nonconcord* or non-complian* or noncomplian*)) ) OR AB ( (guideline* n5 (implement*
nonconcord* or non-complian* or noncomplian*)) )
25 TI ( (guidance n5 (implement* or uptake* or adopt* or adhere* or concord* or complian* or comply or non-adhere* or

nonadhere* or non-concord* or nonconcord* or non-complian* or noncomplian*)) ) OR AB ( (guidance n5 (implement*
26 TI ( (standard? n5 (implement* or uptake* or adopt* or adhere* or concord* or complian* or comply or non-adhere* or

nonadhere* or non-concord* or nonconcord* or non-complian* or noncomplian*)) ) OR AB ( (standard? n5 (implement*
27 TI ( (pathway? n5 (implement* or uptake* or adopt* or adhere* or concord* or complian* or comply or non-adhere* or

nonadhere* or non-concord* or nonconcord* or non-complian* or noncomplian*)) ) OR AB ( (pathway? n5 (implement*
(Continued)
28 TI ( (protocol? n5 (implement* or uptake* or adopt* or adhere* or concord* or complian* or comply or non-adhere* or

nonadhere* or non-concord* or nonconcord* or non-complian* or noncomplian*)) ) OR AB ( (protocol? n5 (implement*
29 22 OR 23 OR 24 OR 25 OR 26 OR 27 OR 28
30 21 AND 29
31 10 OR 18 OR 30
32 ( (MH “Clinical Trials”) OR (MH “Double-Blind Studies”) OR (MH “Randomized Controlled Trials”) OR (MH “Single-
Blind Studies”) OR (MH “Triple-Blind Studies”) ) OR ( TI ( (random* or factorial* or crossover* or cross over* or cross-over*
or placebo* or (doubl* n2 blind*) or (singl* n2 blind*) or assign* or allocat* or volunteer*) ) OR AB ( (random* or factorial*
or crossover* or cross over* or cross-over* or placebo* or (doubl* n2 blind*) or (singl* n2 blind*) or assign* or allocat* or
volunteer*) ) )
33 31 AND 32
34 TI review
37 34 or 35 or 36
38 33 NOT 37
39 (MH “Quasi-Experimental Studies”)
40 TI ( intervention* or multiintervention* or multi-intervention* or postintervention* or post-intervention* or preintervention* or

pre-intervention* ) or AB ( intervention* or multiintervention* or multi-intervention* or postintervention* or post-intervention*
or preintervention* or pre-intervention* )
41 TI ( pre-test* or pretest* or posttest* or post-test* ) or AB ( pre-test* or pretest* or posttest* or “post test* ) OR TI ( preimplement*”
or pre-implement* ) or AB ( pre-implement* or preimplement* )
42 MH Experimental Studies or Community Trials or Community Trials or Pretest-Posttest Design + or Quasi-Experimental

Studies + Pilot Studies or Policy Studies + Multicenter Studies
43 TI ( (comparative N2 study) or (comparative N2 studies) or evaluation study or evaluation studies ) or AB ( (comparative N2

study) or (comparative N2 studies) or evaluation study or evaluation studies )
44 MH “Multiple Time Series” or MH “Time Series”
45 TI pre w7 post or AB pre w7 post
(Continued)
46 TI ( ( quasi-experiment* or quasiexperiment* or quasi-random* or quasirandom* or quasi control* or quasicontrol* or quasi*

W3 method* or quasi* W3 study or quasi* W3 studies or quasi* W3 trial or quasi* W3 design* or experimental W3 method*
or experimental W3 study or experimental W3 studies or experimental W3 trial or experimental W3 design* ) ) or AB ( ( quasi-
experiment* or quasiexperiment* or quasi-random* or quasirandom* or quasi control* or quasicontrol* or quasi* W3 method*
or quasi* W3 study or quasi* W3 studies or quasi* W3 trial or quasi* W3 design* or experimental W3 method* or experimental
W3 study or experimental W3 studies or experimental W3 trial or experimental W3 design* ) )
47 TI ( (time point*) or (period* n4 interrupted) or (period* n4 multiple) or (period* n4 time) or (period* n4 various) or (period* n4
varying) or (period* n4 week*) or (period* n4 month*) or (period* n4 year*) ) or AB ( (time point*) or (period* n4 interrupted)
or (period* n4 multiple) or (period* n4 time) or (period* n4 various) or (period* n4 varying) or (period* n4 week*) or (period*
n4 month*) or (period* n4 year*) )
48 AB ( before* n10 during or before n10 after ) or AU ( before* n10 during or before n10 after )
49 TI time series or AB time series or AB “before-and-after”
50 (MH “Pilot Studies”)
51 TI pilot
52 TI ( collaborativ* or collaboration* or tailored or personalised or personalized ) or AB ( collaborativ* or collaboration* or tailored

or personalised or personalized )
53 (intervention n6 clinician*) or (intervention n6 community) or (intervention n6 complex) or (intervention n6 design*) or

(intervention n6 doctor*) or (intervention n6 educational) or (intervention n6 family doctor*) or (intervention n6 family
physician*) or (intervention n6 family practitioner*) or (intervention n6 financial) or (intervention n6 GP) or (intervention n6
general practice*) Or (intervention n6 hospital*) or (intervention n6 impact*) Or (intervention n6 improv*) or (intervention n6
individualize*) Or (intervention n6 individualise*) or (intervention n6 individualizing) or (intervention n6 individualising) or
(intervention n6 interdisciplin*) or (intervention n6 multicomponent) or (intervention n6 multi-component) or (intervention
n6 multidisciplin*) or (intervention n6 multi-disciplin*) or (intervention n6 multifacet*) or (intervention n6 multi-facet*) or
(intervention n6 multimodal*) or (intervention n6 multi-modal*) or (intervention n6 personalize*) or(intervention n6 person-
alise*) or (intervention n6 personalizing) or (intervention n6 personalising) or (intervention n6 pharmaci*) or (intervention
n6 pharmacist*) or (intervention n6 pharmacy) or (intervention n6 physician*) or (intervention n6 practitioner*) Or (inter-
vention n6 prescrib*) or (intervention n6 prescription*) or (intervention n6 primary care) or (intervention n6 professional*)
or (intervention* n6 provider*) or (intervention* n6 regulatory) or (intervention n6 regulatory) or (intervention n6 tailor*) or
(intervention n6 target*) or (intervention n6 team*) or (intervention n6 usual care)
54 TI ( demonstration project OR demonstration projects OR preimplement* or pre-implement* or post-implement* or postim-

plement* ) or AB ( demonstration project OR demonstration projects OR preimplement* or pre-implement* or post-imple-
ment* or postimplement* )
55 TI ( pre-workshop or preworkshop or post-workshop or postworkshop or (before n3 workshop) or (after n3 workshop) ) or AB

( pre-workshop or preworkshop or post-workshop or postworkshop or (before n3 workshop) or (after n3 workshop) )
56 TI ( trial or (study n3 aim) or “our study” ) or AB ( (study n3 aim) or “our study” )
57 TI ( multicentre or multicenter or multi-centre or multi-center )
(Continued)
58 TI ( (control w3 area) or (control w3 cohort*) or (control w3 compar*) or (control w3 condition) or (control w3 group*)
or (control w3 intervention*) or (control w3 participant*) or (control w3 study) ) or AB ( (control w3 area) or (control w3
cohort*) or (control w3 compar*) or (control w3 condition) or (control w3 group*) or (control w3 intervention*) or (control
w3 participant*) or (control w3 study) )
59 TI ( (time points n3 over) or (time points n3 multiple) or (time points n3 three) or (time points n3 four) or (time points n3
five) or (time points n3 six) or (time points n3 seven) or (time points n3 eight) or (time points n3 nine) or (time points n3 ten)
or (time points n3 eleven) or (time points n3 twelve) or (time points n3 month*) or (time points n3 hour*) or (time points n3
day*) or (time points n3 “more than”) ) or AB ( (time points n3 over) or (time points n3 multiple) or (time points n3 three)
or (time points n3 four) or (time points n3 five) or (time points n3 six) or (time points n3 seven) or (time points n3 eight) or
(time points n3 nine) or (time points n3 ten) or (time points n3 eleven) or (time points n3 twelve) or (time points n3 month*)
or (time points n3 hour*) or (time points n3 day*) or (time points n3 “more than”) )
60 39 OR 40 OR 41 OR 42 OR 43 OR 44 OR 45 OR 46 OR 47 OR 48 OR 49 OR 50 OR 51 OR 52 OR 53 OR 54 OR 55
OR 56 OR 57 OR 58 OR 59
61 31 AND 60
62 TI review
65 62 OR 63 OR 64
66 61 NOT 65
Appendix 5. Cochrane search strategy
#1 MeSH descriptor: [Health Maintenance Organizations] explode all trees
#2 MeSH descriptor: [United States Agency for Healthcare Research and Quality] explode all trees
#3 #1 or #2
#4 MeSH descriptor: [Practice Guidelines as Topic] explode all trees
#5 MeSH descriptor: [Guideline Adherence] explode all trees
#6 #4 or #5
#7 #3 and #6
(Continued)
#8 health maintenance organi?ation* or hmo? or Aetna or Blue Cross Blue Shield Association or CIGNA or Kaiser Permanente
or Humana or Health Net or UnitedHealth Group or Wellpoint or AHCPR:ti (Word variations have been searched)
#9 guideline or guidelines or guidance or standard or standards or pathway or pathways or protocol or protocols:ti (Word variations
have been searched)
#10 #8 and #9
#11 #7 or #10
#13 (guideline* and (implement* or uptake* or adopt* or adhere* or concord* or complian* or comply or non-adhere* or nonadhere*
or non-concord* or nonconcord* or non-complian* or noncomplian*)):ti (Word variations have been searched)
#14 (guidance and (implement* or uptake* or adopt* or adhere* or concord* or complian* or comply or non-adhere* or nonadhere*
#15 (standard* and (implement* or uptake* or adopt* or adhere* or concord* or complian* or comply or non-adhere* or nonadhere*
#16 (pathway* and (implement* or uptake* or adopt* or adhere* or concord* or complian* or comply or non-adhere* or nonadhere*
#17 (protocol* and (implement* or uptake* or adopt* or adhere* or concord* or complian* or comply or non-adhere* or nonadhere*
#18 #12 or #13 or #14 or #15 or #16 or #17
#19 (national near/3 (guideline* or guidance or standard* or pathway* or protocol*)):ti,ab,kw (Word variations have been searched)
#20 (regional near/3 (guideline* or guidance or standard* or pathway* or protocol*)):ti,ab,kw (Word variations have been searched)
#21 (society near/3 (guideline* or guidance or standard* or pathway* or protocol*)):ti,ab,kw (Word variations have been searched)
#22 (association near/3 (guideline* or guidance or standard* or pathway* or protocol*)):ti,ab,kw (Word variations have been
searched)
#23 (academy near/3 (guideline* or guidance or standard* or pathway* or protocol*)):ti,ab,kw (Word variations have been searched)
#24 (board near/3 (guideline* or guidance or standard* or pathway* or protocol*)):ti,ab,kw (Word variations have been searched)
#25 (institute? near/3 (guideline* or guidance or standard* or pathway* or protocol*)):ti,ab,kw (Word variations have been searched)
#26 (ministry near/3 (guideline* or guidance or standard* or pathway* or protocol*))
#27 (department? near/3 (guideline* or guidance or standard* or pathway* or protocol*))
(Continued)
#28 ((health maintenance organi?ation* or hmo? or Aetna or Blue Cross Blue Shield Association or CIGNA or Kaiser Permanente
or Humana or Health Net or UnitedHealth Group or Wellpoint) and (guideline* or guidance or standard* or pathway* or
protocol*)):ti,ab,kw (Word variations have been searched)
#29 MeSH descriptor: [Managed Care Programs] explode all trees
#30 guideline* or guidance or standard* or pathway* or protocol*:ti,ab,kw (Word variations have been searched)
#31 29 and 30
#32 #19 or #20 or #21 or #22 or #23 or #24 or #25 or #26 or #28 or #31
#33 implement* or uptake* or adopt* or adhere* or concord* or complian* or comply or non-adhere* or nonadhere* or non-
concord* or nonconcord* or non-complian* or noncomplian*:ti,ab,kw (Word variations have been searched)
#34 #32 and #33
#35 MeSH descriptor: [Practice Guidelines as Topic] explode all trees
#36 (practice near/3 (guideline* or guidance or standard* or pathway*)):ti,ab,kw (Word variations have been searched)
#37 (clinical near/3 (guideline* or guidance or standard* or pathway* or protocol*)):ti,ab,kw (Word variations have been searched)
#38 #35 or #36 or #37
#40 MeSH descriptor: [Health Plan Implementation] explode all trees
#41 (guideline* near/5 (implement* or uptake* or adopt* or adhere* or concord* or complian* or comply or non-adhere* or
nonadhere* or non-concord* or nonconcord* or non-complian* or noncomplian*)):ti,ab,kw (Word variations have been
searched)
#42 (guidance near/5 (implement* or uptake* or adopt* or adhere* or concord* or complian* or comply or non-adhere* or
searched)
#43 (standard? near/5 (implement* or uptake* or adopt* or adhere* or concord* or complian* or comply or non-adhere* or
searched)
#44 (pathway? near/5 (implement* or uptake* or adopt* or adhere* or concord* or complian* or comply or non-adhere* or
searched)
#45 (protocol? near/5 (implement* or uptake* or adopt* or adhere* or concord* or complian* or comply or non-adhere* or
searched)
(Continued)
#46 #39 or #40 or #41 or #42 or #43 or #44 or #45
#47 #38 and #46
#48 #11 or #18 or #34 or #47
Appendix 6. Proquest search strategy
Set Search
S9 (S1 OR S2 OR S4 OR S6) AND S5Limits applied
S7 (S1 OR S2 OR S4 OR S6) AND S5
S6 ti(practice guideline* OR clinical guideline* OR practice guidance OR clinical guidance OR practice protocol* OR clinical
protocol* OR practice standard* OR clinical standard* OR practice pathway* OR clinical pathway*) AND ab(implement* OR
uptake* OR adopt* OR adhere* OR concord* OR complian* OR comply OR non-adhere* OR nonadhere* OR non-concord*
OR nonconcord* OR non-complian* OR noncomplian*)
S5 ti(random* OR factorial* OR crossover* OR cross over* OR cross-over* OR placebo* OR blind* OR assign* OR allocat* OR
volunteer*) OR ab(random* OR factorial* OR crossover* OR cross over* OR cross-over* OR placebo* OR blind* OR assign*
OR allocat* OR volunteer*)
S4 ab(((national OR regional OR society OR association OR academy OR board OR institute* OR ministry OR department)

near (guideline* OR guidance OR standard* OR pathway* OR protocol*))) AND ab(implement* OR uptake* OR adopt* OR
adhere* OR concord* OR compliant* OR comply OR non-adhere* OR nonadhere* OR non-concord* OR nonconcord* OR
noncomplian* OR noncomplian*)
S2 ti(health maintenance organi?ation* OR hmo? OR Aetna OR Blue Cross Blue Shield Association OR CIGNA OR Kaiser
Permanente OR Humana OR Health Net OR UnitedHealth Group OR Wellpoint OR AHCPR) AND ti(guideline* OR
guidance OR standard* OR pathway* OR protocol*)
S1 ti(((guideline*or guidance OR standard? OR protocol? OR pathways?) AND (implement* OR uptake* OR adopt* OR adhere*
OR concord* OR complian* OR comply OR non-adhere* OR nonadhere* OR non-concord* OR nonconcord* OR non-
complian* OR noncomplian*)))
Appendix 7. WoK search strategy
1 TI=((health maintenance organisation* or health maintenance organization* or hmo or hmos or Aetna or Blue Cross Blue Shield
Association or CIGNA or Kaiser Permanente or Humana or Health Net or UnitedHealth Group or Wellpoint or AHCPR)
) OR AD=((health maintenance organisation* or health maintenance organization* or hmo or hmos or Aetna or Blue Cross
Blue Shield Association or CIGNA or Kaiser Permanente or Humana or Health Net or UnitedHealth Group or Wellpoint or
AHCPR))
2 TI=(guideline* or guidance or standard* or pathway* or protocol*)
3 2 AND 1
4 TI=review
5 TI=(rat or rats or cow or cows or chicken? or horse or horses or mice or mouse or bovine or animal?)
6 5 OR 4
7 3 NOT 6
8 TI=((guideline* and (implement* or uptake* or adopt* or adhere* or concord* or complian* or comply or non-adhere* or
nonadhere* or non-concord* or nonconcord* or non-complian* or noncomplian*)))
9 TI=((guidance and (implement* or uptake* or adopt* or adhere* or concord* or complian* or comply or non-adhere* or
10 TI=((standard? and (implement* or uptake* or adopt* or adhere* or concord* or complian* or comply or non-adhere* or
11 TI=((pathway? and (implement* or uptake* or adopt* or adhere* or concord* or complian* or comply or non-adhere* or
12 TI=((protocol? and (implement* or uptake* or adopt* or adhere* or concord* or complian* or comply or non-adhere* or
13 12 OR 11 OR 10 OR 9 OR 8
14 TS=((regional NEAR/3 (guideline* or guidance or standard* or pathway* or protocol*)))
15 TS=((national NEAR/3 (guideline* or guidance or standard* or pathway* or protocol*)))
16 TS=((society NEAR/3 (guideline* or guidance or standard* or pathway* or protocol*)))
17 TS=((association NEAR/3 (guideline* or guidance or standard* or pathway* or protocol*)))
18 TS=((academy NEAR/3 (guideline* or guidance or standard* or pathway* or protocol*)))
19 TS=((board NEAR/3 (guideline* or guidance or standard* or pathway* or protocol*)))
(Continued)
20 TS=((institute? NEAR/3 (guideline* or guidance or standard* or pathway* or protocol*)))
21 TS=((ministry NEAR/3 (guideline* or guidance or standard* or pathway* or protocol*)))
22 TS=((department? NEAR/3 (guideline* or guidance or standard* or pathway* or protocol*)))
23 TS=((“Managed Care Program*” NEAR/3 (guideline* or guidance or standard* or pathway* or protocol*)))
24 23 OR 22 OR 21 OR 20 OR 19 OR 18 OR 17 OR 16 OR 15 OR 14
25 TS=(implement* or uptake* or adopt* or adhere* or concord* or complian* or comply or non-adhere* or nonadhere* or non-
concord* or nonconcord* or non-complian* or noncomplian*)
26 25 AND 24
27 TS=((practice NEAR/3 (guideline* or guidance or standard* or pathway*))) OR TS=((clinical NEAR/3 (guideline* or guidance
or standard* or pathway* or protocol*)))
28 TS=((guideline* NEAR/5 (implement* or uptake* or adopt* or adhere* or concord* or complian* or comply or non-adhere* or
nonadhere* or non-concord* or nonconcord* or non-complian* or noncomplian*))) OR TS=((guidance NEAR/5 (implement*
nonconcord* or non-complian* or noncomplian*))) OR TS=((standard? NEAR/5 (implement* or uptake* or adopt* or adhere*
noncomplian*))) OR TS=((pathway? NEAR/5 (implement* or uptake* or adopt* or adhere* or concord* or complian* or comply
or non-adhere* or nonadhere* or non-concord* or nonconcord* or non-complian* or noncomplian*))) OR TS=((protocol?
NEAR/5 (implement* or uptake* or adopt* or adhere* or concord* or complian* or comply or non-adhere* or nonadhere* or
non-concord* or nonconcord* or non-complian* or noncomplian*)))
29 28 AND 27
30 29 OR 26 OR 13
31 TS=(random* or blind* or allocat* or assign* or trial* or placebo* or crossover* or cross-over*)
32 31 AND 30
33 32 not 6
34 TI=(intervention*)
35 TS=(((intervention* SAME (clinician* or collaborat* or community or complex or DESIGN* or doctor* or educational or

family doctor* or family physician* or family practitioner* or financial or GP or general practice* or hospital* or impact* or
improv* or individuali*e* or individuali*ing or interdisciplin* or multicomponent or multi-component or multidisciplin* or
multi-disciplin* or multifacet* or multi-facet* or multimodal* or multi-modal* or personali*e* or personali*ing or pharmacies
or pharmacist* or pharmacy or physician* or practitioner* or prescrib* or prescription* or primary care or professional* or
provider* or regulatory or regulatory or tailor* or target* or team* or usual care))))
36 TS=((collaborativ* OR collaboration* OR tailored OR personalised OR personalized))
(Continued)
37 TS=(((demonstration OR pilot) NEXT project*))
38 TI=(pilot)
39 TS=(((pre-post or “pre test*” or pretest* or posttest* or “post test*” or (pre SAME post))))
40 TS=(((pre-workshop or post-workshop or (before SAME workshop) or (after SAME workshop))))
41 TS=((((study SAME aim*) or “our study”)))
42 TS=(((“quasi-experiment*” or quasiexperiment* or “quasi random*” or quasirandom* or “quasi control*” or quasicontrol* or (

(quasi* or experimental) SAME (method* or study or trial or design*)))))
43 TS=(((“time series” SAME interrupt*)))
44 TS=(((time points SAME (over or multiple or three or four or five or six or seven or eight or nine or ten or eleven or twelve or
month* or hour* or day* or “more than”))))
45 TS=((multicentre or multicenter or multi-centre or multi-center))
46 TS=(((control SAME (area or cohort* or compar* or condition or group* or intervention* or participant* or study))))
47 46 OR 45 OR 44 OR 43 OR 42 OR 41 OR 40 OR 39 OR 38 OR 37 OR 36 OR 35 OR 34
48 30 AND 47
49 48 NOT 6
50 7
51 33
52 49
Appendix 8. HMIC search strategy
or Humana or Health Net or UnitedHealth Group or Wellpoint or AHCPR).ti,ab. and (guideline* or guidance or standard*1
(Continued)
7 1 or 2 or 3 or 4 or 5 or 6
protocol*1)).ti,ab
18 8 or 9 or 10 or 11 or 12 or 13 or 14 or 15 or 16 or 17
20 18 and 19
21 clinical guidelines/
(Continued)
24 21 or 22 or 23
25 implementation/
31 25 or 26 or 27 or 28 or 29 or 30
32 24 and 31
33 7 or 20 or 32
34 review.ti.
35 33 not 34
Appendix 9. Trial registers
Trial registers:
Clinicaltrials.gov
(guideline OR guidelines) AND (implement OR implementation Intervention

OR adopt OR adoption OR uptake)
(Continued)
(guideline OR guidelines) AND (concord OR concordance OR Intervention

comply OR compliance OR adherence)
(guideline OR guidelines) AND (implement OR implementation Title

OR adopt OR adoption OR uptake)
(guideline OR guidelines) AND (concord OR concordance OR Title

comply OR compliance OR adherence)
Total:
Duplicates:
Final total:
WHO ICTRP
Guideline OR guide;ines Intervention
guideline OR guidelines Title
Total:
Duplicates:
Final Total:
Appendix 10. Grey literature

• Open Grey (http://www.opengrey.eu/)
• Grey Literature Report (New York Academy of Medicine) (http://greylit.org/)
• Joanna Briggs Institute (http://www.joannabriggs.edu.au/Search.aspx)
• Guideline International Network (GIN) (http://www.g-i-n.net/)
• Agency for Healthcare Research and Quality (AHRQ) Guideline Clearing House (http://www.guideline.gov/) and AHRQ (
www.ahrq.gov/)
• NHS Evidence, who accredit CPG producers within and outside the UK and have access to specialist collections of CPGs (
http://www.evidence.nhs.uk/)
• Scottish Intercollegiate Guideline Network (SIGN) (http://www.sign.ac.uk/)
• Organisations that summarise CPGs (e.g. Map of Medicine (http://www.mapofmedicine.com/), Egton Medical Information
Systems Ltd. (EMIS) (http://www.emis-online.com/)
• eGuidelines for primary care (http://www.eguidelines.co.uk/new˙guidelines.php) (eGuidelines)
• National Institute for Health and Clinical Excellence (NICE) (www.nice.org.uk/); NICE Medicine and Prescribing centre
(MPC) (http://www.nice.org.uk/mpc/index.jsp) (previously the The National Prescribing Centre)
• CMA Infobase (http://www.cma.ca/index.php/ci˙id/54316/la˙id/1.htm)
• SAGE - standards and guideline evidence (http://www.partnershipagainstcancer.ca/2009/02/06/sage-standards-and-guidelines-
evidence/)
Appendix 11. Websites searched
Website name/organisation URL Date Search terms
OpenGrey www.opengrey.eu/ discipline:(06*) AND guideline* AND imple-

ment*
New York Academy of www.greylit.org/

Medicine: Grey Literature Re-
port
GIRAnet - Guideline imple- giranet.org/browse-gitools/ n/a browsed

mentability research and appli-
cation network
International Guideline Library www.g-i-n.net/library/interna- 22/06/2015 Search 1: anything indexed as an implementation

1 tional-guidelines-library tool
International Guideline Library www.g-i-n.net/library/interna- 22/06/2015 Search 2: implement*

2 tional-guidelines-library
Relevant Literature section www.g-i-n.net/working- 23/06/2015 implement*

groups/implementation/imple-
mentation-resources-tools
Development and Training Re- www.g-i-n.net/working- 23/06/2015

sources: Guideline dissemina- groups/implementation/imple-
tion & implementation mentation-resources-tools
Past G.I.N. conferences www.g-i-n.net/conference/ 23/06/2015 n/a browsed

past-conferences
Agency for Healthcare Research www.ahrq.gov/index.html 22/06/2015 (guideline OR guidelines) AND (implement OR
and Quality (AHRQ) implementing OR implementation)
Joanna Briggs Institute www.joannabriggs.org 23/06/2015 implement*
NHS Evidence www.evidence.nhs.uk/ 23/06/2015 implement*
Scottish Intercollegiate Guide- www.sign.ac.uk/ 23/06/2015

line Network (SIGN)
Map of Medicine www.mapofmedicine.com 23/06/2015
Egton Medical Information www.emis-online.com 23/06/2015

Systems Ltd. (EMIS)
Guidelines in practice www.guidelinesinpractice.co. 23/06/2015

uk
(Continued)
Guideline.co.uk www.guidelines.co.uk/ 23/06/2015
NICE www.nice.org.uk/ 23/06/2015
From National Guidelines www.guidelines.gov

Clearinghouse
American Academy of Neurol- (guideline OR guidelines)

ogy AND (implement OR imple-
menting OR implementation)
site: www.aan.com/
American Association of Neu- (guideline OR guidelines)

rological Surgeons AND (implement OR imple-
site: www.aans.org/
American College of Chest (guideline OR guidelines)

Physicians AND (implement OR imple-
site: www.chestnet.org/
American College of Obstetri- (guideline OR guidelines)

cians and Gynecologists AND (implement OR imple-
site: www.acog.org/
American College of Radiology (guide-

line OR guidelines) AND (im-
plement OR implementing OR
implementation) site: www.acr.
org/
American Society for Gastroin- (guideline OR guidelines)

testinal Endoscopy AND (implement OR imple-
site: www.asge.org/
American Urological Associa- (guideline OR guidelines)

tion Education and Research, AND (implement OR imple-
Inc. menting OR implementation)
site: www.auanet.org/
British Committee for Stan- (guide-

dards in Haematology line OR guidelines) AND (im-
implementation) site: www.bc-
shguidelines.com/
(Continued)
Cancer Care Ontario (guideline OR guidelines)

AND (implement OR imple-
site: www.cancercare.on.ca/
CancerControl Alberta (guide-

implementation) site: www.al-
bertahealthservices.ca
Centers for Disease Control (guideline OR guidelines)

and Prevention AND (implement OR imple-
site: www.cdc.gov/
Cincinnati Children’s Hospital (guideline OR

Medical Center guidelines) AND (implement
OR implementing OR imple-
mentation) site: www.cincin-
natichildrens.org
Congress of Neurological Sur- (guide-

geons line OR guidelines) AND (im-
implementation) site: www.cns.
org/
European Academy of Neurol- (guide-

ogy line OR guidelines) AND (im-
implementation) site: www.ea-
neurology.org/
European Association of Urol- (guide-

ogy line OR guidelines) AND (im-
implementation) site: uroweb.
org/
Hartford Institute for Geriatric (guideline OR guidelines)

Nursing AND (implement OR imple-
site: www.hartfordign.org/
Institute for Clinical Systems (guide-

Improvement line OR guidelines) AND (im-
(Continued)
implementation) site: www.icsi.

org/
Michigan Quality Improve- (guideline OR guidelines)

ment Consortium AND (implement OR imple-
site: www.mqic.org/
National Clinical Guideline (guideline OR guidelines)

Centre AND (implement OR imple-
site: www.ncgc.ac.uk/
New York State Department of (guideline OR guidelines)

Health AND (implement OR imple-
site: www.health.ny.gov/
Ontario Ministry of Health and (guideline OR guidelines)

Long-Term Care AND (implement OR imple-
site: www.health.gov.on.ca/en/
Program in Evidence-based (guideline OR guidelines)

Care AND (implement OR imple-
site: www.cancercare.on.ca
Royal College of Nursing (guide-

implementation) site: www.rcn.
org.uk
Royal College of Obstetricians (guideline OR guidelines)

and Gynaecologists AND (implement OR imple-
site: www.rcog.org.uk/
Society of Obstetricians and (guide-

Gynaecologists of Canada line OR guidelines) AND (im-
implementation) site: sogc.org/
U.S. Preventive Services Task (guideline OR guidelines)

Force AND (implement OR imple-
site: www.uspreventiveservices-
taskforce.org/
(Continued)
University of Michigan Health (guideline OR guidelines)

System AND (implement OR imple-
site: www.uofmhealth.org/
CONTRIBUTIONS OF AUTHORS
GF contributed to the writing of the protocol, led the screening of the studies for inclusion and data extraction, and drafted the review.
AH assisted with screening studies for inclusion and data extraction, and commented on drafts of the review.
LG assisted with screening studies for inclusion and data extraction, and commented on drafts of the review.
ME contributed to the writing of the protocol.
JG commented on the final version of the review prior to peer review.
GL suggested the topic of the review, and commented on drafts of the review.
SS contributed to the writing of the protocol, assisted with screening studies for inclusion, and commented on drafts of the review.
DECLARATIONS OF INTEREST
GF None known.
AH None known.
LG None known.
ME None known.
JG holds the Canada Research Chair in Health Knowledge Transfer and Uptake.
GL is Deputy Chief Executive and Director of Health and Social Care at the National Institute for Health and Care Excellence (NICE),
and has responsibility for the implementation programme that includes the development of implementation tools.
SS None known.
SOURCES OF SUPPORT
Internal sources
• Nuffield Department of Population Health, University of Oxford., UK.
External sources
• NIHR Cochrane EPOC programme grant, UK.
DIFFERENCES BETWEEN PROTOCOL AND REVIEW

Two new review authors (AH and LG), who were not involved at the protocol stage, are included in the review team.

Flodgren 2016

Uploaded by

Copyright:

Available Formats

Flodgren 2016

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Flodgren 2016

Uploaded by

Copyright:

Available Formats

Cochrane Database of Systematic Reviews

Tools developed and disseminated by guideline producers to

Tools developed and disseminated by guideline producers to

Editorial group: Cochrane Effective Practice and Organisation of Care Group.

PLAIN LANGUAGE SUMMARY

Outcomes M edian ARD No of Participants Certainty of the evi- Comments

BACKGROUND system). Learning modules are a popular approach to support-

Description of the intervention How the intervention might work

A secondary objective is to determine which approaches to guide- i) Tailoring

Searching other resources

Electronic searches Trial registries

Patients i) Interventions targeting the healthcare professional

ii) Interventions targeting the patient Mode of delivery:

The randomisation sequence and the allocation concealment were

Healthcare resource use and costs

Characteristics of included studies [ordered by study ID]

Methods Study design: Cluster-RCT

Participants Participating providers: Physiotherapists n = 68 practices (113 physiotherapists); In-

Outcomes Main outcome:

Notes Ethical approval and informed consent obtained (yes/no): Yes

Bias Authors’ judgement Support for judgement

Random sequence generation (selection Low risk p. 108, Col. 1, Para. 2

Blinding of outcome assessment (detection Low risk p.108, Col.1, Para.5

Baseline characteristics similar Unclear risk Physiotherapists in the intervention group

Other bias Low risk No evidence of other risk of bias

Methods Study design: Cluster-RCT

Participants Participating healthcare professionals: Physicians; n = 704; Intervention (Dual Inter-

Interventions Aims: To compare the (independent) and combined effectiveness of 2 implementation

Outcomes Main outcome:

Notes Ethical approval and informed consent obtained (yes/no): No information

Bias Authors’ judgement Support for judgement

Random sequence generation (selection Low risk p. 433, Col. 2, Para. 1

Allocation concealment (selection bias) Low risk Cluster-RCT .

Blinding of outcome assessment (detection Unclear risk p. 433, Col. 2, Para. 3

Baseline outcome measures similar Unclear risk No baseline measure of outcome

Other bias Low risk No evidence of other risk of bias

Methods Study design: Cluster-RCT

Interventions Aims: To determine whether implementation of an evidence-based guideline would

Outcomes Main outcomes:

Notes Ethical approval and informed consent obtained (yes/no): Yes

Bias Authors’ judgement Support for judgement

Random sequence generation (selection Unclear risk No information

Baseline characteristics similar Low risk Baseline characteristics similar (Table 2)

Baseline outcome measures similar Unclear risk No baseline measures of outcome

Other bias Low risk No evidence of other risk of bias

Participants Participating providers:

Interventions Aims: To evaluate the effectiveness of an educational toolkit focusing on cardiovascular

risk reduction strategies

Outcomes Main outcomes:

Bias Authors’ judgement Support for judgement

Allocation concealment (selection bias) Low risk See above

Other bias Unclear risk Endocrinologist in Ontario also received

ACEI/ARB: angiotensin converting enzyme inhibitor/angiotensin receptor blocker

Characteristics of excluded studies [ordered by study ID]

Study Reason for exclusion

Eccles 2002 C-RCT: Tools not developed by existing guideline developers

Eccles 2007 C-RCT.Tools not developed by existing guideline developers

Flottorp 2002 Tools not developed by existing guideline developers

Fretheim 2006 Tools not developed by existing guideline developers

17 (national adj3 (guideline1 or guidance or standard1 or pathway1 or protocol1)).ti,ab

18 (regional adj3 (guideline* or guidance or standard1 or pathway1 or protocol*1)).ti,ab

19 (society adj3 (guideline* or guidance or standard1 or pathway1 or protocol*1)).ti,ab

20 (association adj3 (guideline* or guidance or standard1 or pathway1 or protocol*1)).ti,ab

21 (academy adj3 (guideline* or guidance or standard1 or pathway1 or protocol*1)).ti,ab

22 (board adj3 (guideline* or guidance or standard1 or pathway1 or protocol*1)).ti,ab

23 (institute? adj3 (guideline* or guidance or standard1 or pathway1 or protocol*1)).ti,ab

24 (ministry adj3 (guideline* or guidance or standard1 or pathway1 or protocol*1)).ti,ab

25 (department? adj3 (guideline* or guidance or standard1 or pathway1 or protocol*1)).ti,ab

32 (practice adj3 (guideline1 or guidance or standard1 or pathway*1)).ti,ab

33 (clinical adj3 (guideline1 or guidance or standard1 or pathway1 or protocol1)).ti,ab

64 (“quasi-experiment” or quasiexperiment or “quasi random” or quasirandom or “quasi control” or quasicontrol or ((quasi*