Introduction to Evidence-Informed Decision Making

Developed by Donna Ciliska, RN, PhD

Professor, McMaster University, and
Scientific Director, National Collaborating Centre for Methods and Tools
Introduction to Evidence-Informed Decision Making
How do I use this learning module?
Estimated total time: 5 hours

Objective: To understand the components of evidence-informed decision-making.

Process: This module is built on a scenario that will allow you to understand and apply each
stage of the Evidence-Informed Decision Making process. Once you read the scenario, you can
go to any section. (Time estimates are in brackets).

Links: Each time you see the word scenario, it is linked to the actual scenario and will take you
there if you click on it. Similarly, the key terms are linked to a definition in a glossary.

1. What is Evidence-Informed Decision making? Why bother? (0.5 hours)

2. Ask. How do I frame the question? (0.5 hours)

3. Acquire. How can I find the best evidence in five minutes or less? (1 hour)

4. Appraise. How can I decide if the particular study is good enough to apply? (0.5 hours)

5. Integrate. How do I decide which of multiple studies to use? (0.5 hours)

6. Adapt. How do I use the information from #5 in decision-making/policy brief? (1 hour)

7. Apply. How do I develop the implementation the plan? (0.5 hours)

8. Analyze. How do I evaluate if the plan worked? (0.5 hours)

Scenario:

Obesity is on the rise in Canada. You have been invited to be part of a national panel of
practitioners, researchers, and program-level and government-level policy makers drawn
together for a “think tank” to come up with ‘evidence-informed’ recommendations for the
prevention of obesity in children. Your own expertise is related to school-based programs for
children in primary schools (roughly 5-13 yrs old). You have heard the words ‘evidence-
informed decision-making’ but, before you go to the panel, you want to be sure you understand
the process. You find this on-line educational module to help you!
Part 1. What is Evidence-Informed Decision making? Why
bother?
(0.5 hours)

Consider:

Billions of dollars are spent annually on health-related research

It takes approximately fifteen years to get research into recommended policy
and 40% implementation in practice (Antman et al, 2001)

30-50% of people received recommended care (Schuster et al, 1998)

30-40% of patients do not get treatments of proven effectiveness (Grol, 2001)

20-30% of patients received acute care that was not needed or that was
potentially harmful (Schuster et al, 1998)

Is there any reason to believe it is any different in your own area of expertise?

These statistics give you some perspective on the current

research ► practice gap

that exists. The evidence-based medicine (EBM) movement was born of the realization of the
gap, and began an effort to bridge the
research ► practice

and

research ► policy gaps.

A brief history:
EBM as a term was first coined by Gord Guyatt in 1992, and a commonly used definition was
offered by Dave Sackett and colleagues:

“the conscientious, explicit and judicious use of the current best evidence in
making decisions about the care of individual patients” (Sackett, 1996)

The Evidence-Based Medicine Work Group, under the leadership of Guyatt, published a series
of 25 articles for the Journal of the American Medical Association (JAMA) between 1993 and
2000 that outlined criteria to evaluate current evidence to support clinical decisions. These
Users’ Guides have formed the basis of most of the critical appraisal tools that exist and were
revised and published together in book form (Guyatt & Rennie, 2002). EBM was recently
named one of the top ten most important innovations in health care in the last 150 years, in a
poll held by the British Medical Journal (BMJ), arguing that EBM allows the other innovations
to be implemented.

EBM has expanded to include many disciplines such as evidence-based nursing, veterinary
medicine, dentistry, policy-making and pastoral care. In every case, evidence-based healthcare
involves practitioners or policy makers using their expertise to combine the best available
evidence, knowledge of available resources with patient or population circumstances, values
and preferences in decision-making.

How could you argue with the goal of bridging that

research ► practice gap?

There has been a backlash to EBM related to the strongly held myth that the only acceptable
evidence is from randomized controlled trials or meta-analyses. We have begun to use the term
‘Evidence-Informed Decision Making’ (EIDM) to attempt to get beyond some resistance to EB
practice, and to connote that other types of evidence are useful in making decisions.

In your reading of other material, you will see many other terms such as knowledge translation
or knowledge exchange. In a project to set up search terms to find this literature, Ann
McKibbon and colleagues at McMaster University have come up with over 60 terms for this
idea of getting research evidence into practice and policy! Just think of them as terms related to
this enterprise of getting the research evidence into decision-making.

Practice!
Take a few minutes to jot down some answers to the following questions.

1. What are the barriers for you in getting research evidence into your work (whether
patient care, program design, policy design or research study design)?

2. What are the blocks that have to do with you?

3. What are the blocks that have to do with your job design?

4. What are the blocks that have to do with your organization?

5. What are the blocks that have to do with your community?

Barriers:
Much research has gone into studying the barriers to using research evidence in practice. Here
are the big ones:

Time

Inability to access research

Inability to understand the language of research

Lack of critical appraisal skills

Lack confidence in making change based on research evidence

Lack of sense of control over practice

Culture – resistance to change, decisions based on history

Lack of organization valuing or supporting evidence-based practice

Lack of consensus on what constitutes evidence.

NOTE: many hours have been spent debating what constitutes evidence. There are many
different definitions from many different disciplines. In the context of this learning module, the
term ‘research evidence’ will be used when discussing studies. Otherwise, ‘evidence’ will
connote the broader definition, including other forms of knowing.

A few studies have looked at characteristics that influence whether or not a change in practice
happens as a result of new research evidence. They have to do with issues related to the:

Individual – e.g., age, education, years in same position

Organization – e.g., size, complexity, research done in that organization, affiliation with
a university

Environment – e.g., rural/urban, university or research centre in geographic area.

Evidence itself – e.g., complexity, difference from current practice, resource

requirements.

So we are beginning to know what is blocking us from putting evidence into practice. How do
we get beyond those barriers? There are some active areas of research, today, on how to
change practice and less on how to get research evidence into policy. More on that in section 7,
“Apply”.

How does knowledge get research evidence into practice and policy?
- the CIHR (2007) Knowledge to Action Framework

Graham and colleagues developed a framework entitled “Knowledge to Action” to integrate

knowledge creation and application (Graham et al., 2006). They described a knowledge
“funnel” where knowledge is created, aggregated, distilled and tailored into clear and practical
products for use in the field. Central to the Knowledge to Action Process is that, at each phase
of knowledge creation, producers customize knowledge products to the unique needs of
potential users. A process of collaboration throughout the process, including the question
phase, has the potential to make the research more meaningful and useable for practitioners,
managers and policy developers.

The action cycle circles the knowledge creation funnel. The action cycle contains 7 activity
phases that are recommended to support the “translation” of knowledge to users. The activities
are taken from a review of 60 theories of planned action (Graham et al 2006).

The model recognizes the nonlinear process of knowledge translation: the authors expect that
each action phase can be influenced by the preceding action phases, and by feedback between
the phases.

You will see that this model fits with the steps of Evidence-informed decision-making. As a
clinician, manager or policy-maker, you will be starting with the box at the bottom of the
diagram – identifying a problem, identifying, reviewing and selecting knowledge, then moving
to the left on the circle, adapting the knowledge to the local context, considering barriers to the
implementation, selecting interventions, monitoring their use and evaluating outcomes. The
final stage feeds back to where we started –with the identification of problems.
 OTHER RESOURCES

Canadian Health Services Research Foundation. Promising Practices in Research Use.

http://www.chsrf.ca/promising/index_e.php
You will find a growing repository of papers related to people, processes and structures that
could help healthcare organizations make better use of evidence.

Canadian Institutes of Health Research (CIHR). About Knowledge Translation.

http://www.cihr-irsc.gc.ca/e/29418.html
Provides definitions, further elaboration of the ‘knowledge cycle” and examples of CIHR
initiatives.

Centre for Evidence-Based Medicine, Oxford. What is EBM? http://www.cebm.net/?o=1014

References

Antman, E.M., Lau, J., Kupelnick, B., Mosteller, F., Chalmers, T.C. (2001). Acomparison of
results of meta-analyses of randomized control trials and recommendations of clinical
experts. Treatments for myocardial infarction. JAMA. 1992 Jul 8;268(2):240-8.

Graham, I., Logan, J., Harrison, M.B., Straus, S.E., Tetroe, J., Caswell, W., & Robinson, N.
(2006). Lost in Translation: Time for a Map? The Journal of Continuing Education in
the Health Professions, 26(1), 13-24.

Grol, R. (2001). Successes and failures in the implementation of evidence-based guidelines for
clinical practice. Medical Care, 39, 8 (S2), II46-54.

Guyatt, G. & Rennie, D. (Eds) (2002). Users’ Guides to the Medical Literature: A manual for
Evidence-Based Clinical Practice. American Medical Association.

Sackett. D.L., Rosenberg, W.M., Muir-Gray, J.A., Haynes, R.B., & Richardson, W.S. (1996).
Evidence-based medicine: what it is and what it isn’t. BMJ, 328, 535-536

Schuster, M.A., McGlynn, E.A., & Brook, R.H. (1998). How Good Is the Quality of Health
Care in the United States? Milbank Quarterly, 76 (4), 517-563
Part 2. Ask. How do I frame the question?
(0.5 hours)

The first stage of evidence-informed decision-making is to frame the question. Using the
scenario about developing recommendations regarding prevention of obesity in children, you
decide to update your own knowledge.

1. You search PubMed on obesity and get 17,826 hits. Not a chance you are going there!

2. You try to limit your search by adding children, resulting in 15, 238 hits. Still no way
you can get through those. Most people give up at this stage, and decide to go with
whatever studies they have in their personal files.

Practice, research and policy questions can be of many types – incidence/prevalence; burden of
suffering and quality of life, experience and meaning of illness, causation, harm, assessment
(or diagnosis), prognosis, costs. Frequently, however, most of us are concerned with
effectiveness of therapy or interventions.

When considering intervention or therapy questions, you can use the following PICO format
to help you be very specific about your question. This also helps to formulate your search of
the research evidence.

P atient / P opulation
Who are the people involved –individuals, families, populations? Age? Specific
problem or prevention issue?

I ntervention
What specific preventive, therapeutic, health services strategies are you considering?

C omparison
What is current ‘usual care’ or ‘usual circumstance’ or ‘standard of care’?

O utcome
What are the patient or population level outcomes of most significance?

When looking at context or experiences, these questions are best answered by qualitative
research, so the P S question frame is used:

P atient / P opulation
Who are the people involved –individuals, families, populations? Age? Specific
problem or prevention issue?

S ituation
What circumstances or experiences do you want to know about?
Practice!
1. Using the scenario about developing recommendations regarding prevention of obesity in
children, you decide to update your own knowledge related to school-based interventions.
How will you frame the question? Jot down how you would apply the PICO format before
going on to the next page:

P atient / P opulation

I ntervention

C omparison

O utcome

2. From your answers, compose a focused question with all the above parameters.

3. Possible answer:

P atient / P opulation: school children, aged 5-12

I ntervention addition of a daily physical activity program

C omparison usual health and physical education curriculum alone

O utcome rate of obesity, rate of overweight, mean body-mass index (BMI),

mean weight for height

So you might write the question as:

What is the effectiveness of daily physical activity programs done in the

schools, compared to the usual health and physical education curricula
alone, on the rate of obesity in school children?
4. A related qualitative question might be:

P atient / P opulation obese school children

S ituation school physical activity program

You might want to know if obese children are teased or ridiculed by other children or teachers
when they participate in physical activity programs. Do they experience discrimination? Are
they made to feel inadequate?

One possible qualitative question (phenomenology) would be:

What is the experience of school children when they participate in

physical activity programs at school?

OTHER RESOURCES

Duke University Medical Center Library and Health Sciences Library, UNC-Chapel Hill. The
well-built clinical question.
http://www.hsl.unc.edu/Services/Tutorials/EBM/Question.htm

References:

Fineout-Overholt, E., & Johnston, L. (2005). Teaching EBP: Asking searchable, answerable
clinical questions. Worldviews on Evidence-Based Nursing, 2, 157-160.
Part 3. Acquire. How can I find the best evidence in 5 minutes
or less?
(0.5 hours)

Using the question you framed in Part 2 –

What is the effectiveness of daily physical activity programs done in the schools, compared to
the usual health and physical education curricula alone, on the rate of obesity in school
children?
– how would you begin a search to get yourself the best evidence?

As mentioned in Part 2, just searching PubMed on obesity or obesity in children yielded over
15,000 hits. One of the downsides of the information age is clearly information overload!

A hierarchy of quantitative evidence has been proposed to indicate that some study designs are
considered stronger, that is, more free of bias, than others. No study of people, designed by
people, could ever be totally free of bias, but as you go down the hierarchy, there is a
likelihood of greater bias. The use of this hierarchy has led to one of the unfortunate myths of
evidence-informed decision making, that only systematic reviews or randomized controlled
trials can be considered evidence. The reality is that all study designs are forms of evidence.
When you are making a decision about instituting a change in practice or policy, however, you
would want to use the evidence from the highest point in the hierarchy that you could find. For
some questions, that may very well be case control studies or expert opinion!

Hierarchy of quantitative evidence

Systematic reviews of randomized controlled trials
Randomized controlled trials
Systematic reviews of nonrandomized or cohort studies
Cohort studies
Case control studies
Case series
Case report
Ideas, editorials, opinions
Animal research
Invitro (test tube) research

▲ It is very important to note that this hierarchy does NOT include qualitative research.
Qualitative research answers questions about experience or meaning. It is a parallel to the
hierarchy of quantitative evidence.

Brian Haynes has described a pyramid (the 5S Pyramid), for finding the best evidence with the
least amount of time and effort (Haynes, 2007).
 Search

systems

summaries
synopses

synthesis
studies

The 5S Pyramid
Adapted from B. Haynes (2007).

Systems are electronic systems that might be so sophisticated as to be linked to patient records
and prompt practitioners regarding guidelines for care (e.g., what tests to order or what
interventions need to be done). For example, for a patient with type II diabetes, it would
prompt the care-giver that blood work, eye exam, foot exam and diet review need to be done.

The systems level also includes guidelines. You can access guidelines from sites such as the
National Guidelines Clearinghouse (www.guidelines.gov) or the Registered Nurses’
Association of Ontario Best Practice Guidelines (www.rnao.org). Many of the guidelines are
free full-text.

Summaries are usually text-based and are related to a specific disease or condition. An
example includes Clinical Evidence (www.clinicalevidence.com). Unfortunately, you must
subscribe either individually or via an institution.

Synopses are brief reports (1-2 pages) of pre-appraised individual studies or systematic
reviews which give key methodological details and results, along with an expert commentary
on issues of applying the results in practice. Examples of synopses are found in the evidence-
based journals (eg: Medicine, Nursing, Dentistry, Health Policy). There are currently 23 such
journals. For example, see Evidence-Based Nursing (www.evidencebasednursing.com) or
Evidence-Based Dentistry (http://www.nature.com/ebd/index.html). Some of these are open
access, whereas others require a subscribtion, either individually or via an institution.

Syntheses include systematic reviews of all studies that could be found on a particular focused
question. These include The Cochrane Library (www.chochrane.org), the Agency for
HealthCare Research and Quality (AHRQ) Evidence-based Practice Centre Reviews
(http://www.ahrq.gov). Both groups have similar, rigorous methods for review.

Another source for systematic reviews and summaries about interventions related to public
health in Canada is the Effective Public Health Practice Project
(http://old.hamilton.ca/phcs/ephpp/ReviewsPortal.asp). Further, Health-Evidence.ca
(http://www.health-evidence.ca) rates reviews that are relevant to public health in Canada,
summarizes them and provides recommendations for practice and policy that arise from the
reviews.

Studies are individual studies related to a particular focused question. There are several
searchable databases that would help you find individual studies. The most used in Canada
include Medline, PubMed and CINAHL, but there are many other specialized databases.
PubMed is good to know as it is a free-access version to Medline, found at
http://www.ncbi.nlm.nih.gov/sites/entrez?db=pubmed). PubMed provides a full citation and
abstract, as well as links to any full-text articles that are free to the public.

As mentioned in Part 2, when we searched PubMed for any type of study related to obesity in
children, we came up with over 15,000!

Exploring search strategies

Using the search Pyramid above, start any search from the top.

Back to the question from our scenario:

What is the effectiveness of daily physical activity programs done in the schools, compared to
the usual health and physical education curricula alone, on the rate of obesity in school
children?

1. Systems:

Most of us do not have sophisticated systems where we work but we can get free on-line access
to guidelines. If you find a well-done guideline that was recently published, you might start and
end your search there!

Open up another window in your browser so that you can keep this window open and search
for some information at the same time. In your 2nd window, go to
www.guidelines.gov
Guidelines are usually done for broader questions, such as “prevention of obesity in children”,
or “treatment of obesity in children” rather than focused on something like “daily physical
activity”. So, you type on the search box “obesity and children”. You will get over 100 “hits”
or references – a bit too much to sort through! Your next step would be to use their “Detailed
Search” that prompts you for keywords, conditions, age. As those come up, try the following
limits.

Keyword: physical activity

Disease/Condition: obesity
Treatment/Intervention: physical activity
Age Range: Adolescent (13 to 18 years), Child (2 to 12 years)
Sort Order: Relevance

What you get now is between 10-20 hits. Much more manageable!
Here are a few titles:

Increasing physical activity in schools: kindergarten through eighth grade.

University of Iowa Gerontological Nursing Interventions Research Center, Research
Translation and Dissemination Core - Academic Institution. 2005 May. 42 pages.
NGC:004518

Primary prevention of childhood obesity. Registered Nurses Association of Ontario -

Professional Association. 2005 Mar. 88 pages. NGC:0042

Active healthy living: prevention of childhood obesity through increased physical

activity. American Academy of Pediatrics - Medical Specialty Society. 2006 May
1. 9 pages. NGC:004962 65

Overweight in children and adolescents: pathophysiology, consequences,

prevention, and treatment. American Heart Association - Professional Association.
2005 Apr 19. 14 pages. NGC:004277

All look like they have potential, but the 1st and the 3rd actually have physical activity in their
title, so you will check those out first.

2. Synopses
One barrier to using evidence in decision-making is lack of skills in critical appraisal. Synopses
are available where different groups, like the McMaster Health Knowledge Refinery,
(http://hiru.mcmaster.ca/hiru/HIRU_McMaster_HKR.htm), have gone to all the work of
finding and critically appraising articles, including reviews and single studies.

For public health policy and practice questions, one of the best sources of information is from
Health-Evidence.ca (www.health-evidence.ca), which is described in more detail with practice
tips under syntheses. Staff conduct a thorough search for reviews, then critically appraise the
and rate them as strong, moderate or weak. Strong reviews get a 2 page synopsis – a summary
of key methods issues with recommendations for practice and policy.

Other sources of synopses are the evidence-based journals. Unfortunately, you need to belong
to a consortium through your local university or hospital or government system to access those.
However, these journals are a fantastic source of ‘pre-appraised’ information. The staff rate
every study in a wide range of healthcare journals using pre-set quality criteria, select only the
best quality studies and summarize those studies in a 1 or 2 page abstract with the implications
added by an expert in the field. For policy makers, Evidence-Based Health Policy and
Management and Evidence-Based HealthCare deals with more policy and systems issues,
where as Evidence-based Dentistry or the Evidence-Based Nursing are obviously more
discipline focused, and Evidence-Based Mental Health is more condition-focused.
Practice!
If you have access to a library consortium, search a few of the evidence-based journals for a
synopsis, review, or single study that answers the question posed in Part 2:
What is the effectiveness of daily physical activity programs done in the schools,
compared to the usual health and physical education curricula alone, on the rate
of obesity in school children?

One example found that was relevant for the scenario is:

Kerr, C.M. (commentator) (2000). A school based, interdisciplinary

curriculum in grades 6 and 7 reduced obesity in girls. Evidence-Based
Nursing, 3: 13.

Note that the title gives you the conclusion of the study.

3. Syntheses

Syntheses may be summarized into a synopsis, as discussed in step 2 above. However, they
involve a group of people doing a thorough search on a particular therapy for a particular
problem. There are many different terms used, but, most often, a systematic review means that
the review team undertook a very broad and in-depth search for published and unpublished
studies, had at least two people independently rate relevance and quality and do data extraction
for the primary studies, then put together the results in a meaningful way that goes beyond just
describing each study in detail. A systematic review must be done in order to get to a meta-
analysis, where, if appropriate, statistical results from individual studies are combined to give
an estimate of the overall effect.

The Cochrane Library (http://www.cochrane.org/reviews/) is a prime source for syntheses.

Review groups from around the world follow a rigorous process to find, critically appraise,
then synthesize the results from all the relevant studies. (See the Cochrane Collaboration
http://www.cochrane.org). The Cochrane Library is a searchable database of these high quality
reviews. There is currently free access in only a few provinces in Canada, but efforts are
underway to make it freely accessible across Canada, so keep checking if you do not have
access through a library service. There are several different sections of the library:
 1. systematic reviews
2. protocols (Someone has developed the plan for a systematic review on the topic. An
expected completion date is usually given.)
3. DARE (Database of Abstracts of Reviews of Effects) – a database of other reviews that
have not been done by Cochrane groups

When you look at a review in the Cochrane Library, you first get a structured abstract,
followed by the “plain language statement”, which is useful if you do not understand the
language of research.

Practice!
If you do have access, go to the Cochrane Library and to the search box. Explore what you get
when you type in obesity; or physical activity and school . What did you find that is relevant to
our scenario?

Several come up – more specific ones with the latter search. For example, this one looks
useful:

Summerbell, C.D., Waters, E., Edmunds, L.D., Kelly, S., Brown, T., & Campbell K.J.
Interventions for preventing obesity in children. Cochrane Database of Systematic Reviews
2008 Issue 2.

FYI: The results of this review indicate that several studies were found. The authors divided
the results by short versus long-term (more than 12 months), and by the type of intervention
(diet plus activity versus either alone). Two long-term studies of physical activity alone
showed that children were more likely to have a lower BMI after a physical activity program,
but the results were larger if diet and physical activity were combined and the effect was
greater in girls than boys.

Health-Evidence.ca

As noted in the synopses above, for public health policy and practice questions, one of the best
sources of information is from Health-Evidence.ca (www.health-evidence.ca). Using a very
thorough search, the team broadly searches every six months for any new reviews. They
critically appraise the review and rate it as strong, moderate or weak, then provide a two-page
summary of key methods issues, with recommendations for practice and policy.

Keeping this page available, open a 2nd window in your browser and click on www.health-
evidence.ca

Here is what you will see:

Practice!

1. Click the top right where the magnifying glass says “SEARCH articles”. It leads to a refined
search page, where you will see Focus. There, you can specify physical activity and nutrition
(with subfields like healthy weight); Intervention Location where you can indicate school, and
age fields to check. Several reviews come up; all have quality ratings and some have the
summary with recommendations in a pdf document. This is a gold mine where all the
‘panning’ has been done for you. If the particular review has a summary (same as the
synopsis), you get recommendations for policy and practice, available in English or French.

2. Explore how the results changes with different search box fields.

Of interest, the same systematic review comes up as identified in The Cochrane Library search
above:
Summerbell, C.D., Waters, E., Edmunds, L.D., Kelly, S., Brown, T., & Campbell
K.J. Interventions for preventing obesity in children. Cochrane Database of
Systematic Reviews 2008 Issue 2.

However, here the review has already been critically appraised for you (pre-appraised) and
rated to be strong. In addition, there is a summary of the study and implications for practice
and policy that is located in an attached pdf document.

4. Studies

Sometimes the topic of your interest is one that has some individual studies, but no one has
made the commitment to pull them together into a systematic review. At the other extreme, you
may be searching for an intervention that is so novel, there have been no published studies.

For individual studies, there are several possible databases that you could check: Medline,
CINAHL and other more specialized databases such as CancerLit. This unit will focus on
PubMed (public access Medline) as it is a free-access searchable database that will even give
you full-text provided the journal where the paper is published has allowed that. Otherwise,
you at least usually get the abstract, which will give the idea if you want to go further to
retrieve the paper. PubMed allows you to search for reviews as well as individual studies.

Open PubMed now in another browser window.

www.pubmed.gov Here is the opening window in PubMed:
a) Using “Clinical Queries”

This PubMed search option has a powerful, tested, built-in and lengthy search strategy that
allows you to find answers to your questions quickly. What follows is a practice run at using
this powerful tool.

Down the shaded left side box on the opening page above, you will see “PubMed Services”.
There, click on the section that says “Clinical Queries”. From your actual open web browser
or the screenshot below, you can see that there are several options, including “Search for
Clinical Studies by Category” and “Find Systematic Reviews”.

Following the hierarchy, you go to the “Find Systematic Reviews” option and type in
school interventions to prevent obesity and get approximately ten hits – all systematic reviews.
You can quickly scan and find, from the titles, that one is for 0-5 years, one for preschool and
another is for adolescents. The other 6 are relevant and have been published between 1993 and
2007. Interestingly, the same systematic review comes up here, as in The Cochrane Library and
in Health-Evidence.ca:

Summerbell, C.D., Waters, E., Edmunds, L.D., Kelly, S., Brown, T., & Campbell K.J.
Interventions for preventing obesity in children. Cochrane Database of Systematic Reviews
2008 Issue 2.
If you do not find a good systematic review, return to the Clinical Queries section and, under
Search for Clinical Studies by Category, click the circle on Therapy. Type in the box some
terms such as physical activity and obesity prevention and children. You get approximately 40
studies. For most people making clinical and policy decisions, this number is probably not
manageable!

b) Using Special Queries

Of note, other types of studies get (mis)classified for easier searching in PubMed. Back to the
opening page, click on Special Queries. Here you will find Health Services Research (HSR)
Queries, which would be of special interest to managers and policy-makers. Once again, click
there and you will see there are several options, such as:
o economics
o outcome assessment
o process assessment
o qualitative research NOTE – a strange place to put qualitative research as
most qualitative research is not about health services research, and most people
looking for answers to qualitative questions would not think to look under
health services research.
c) Using PubMed Limits

If you have no luck with the Clinical Queries or Special Queries, another option is to use
Limits in PubMed. Back to the opening page, you can type in the words to describe your topic,
then click on limits just below the search box. You will see that you can limit by several factors
– author, year, type of article, age. Play with using the same terms in the box, but changing the
limits to see what you get.

OTHER RESOURCES

Centre for Evidence-Based Medicine. EBM tools. Finding the best evidence.
http://www.cebm.net/index.aspx?o=1038

Forrest, J.L. (2001). Enhancing your practice through evidence-based decision-making: finding
the best evidence. Journal of Evidence-Based Dental Practice, 1, 127-136.

The McMaster Health Knowledge Refinery. A collection of projects related to retrieval,

appraisal, organization, classification, dissemination and uptake of evidence from research that
is relevant to health care. http://hiru.mcmaster.ca/hiru/HIRU_McMaster_HKR.htm

National Library of Medicine. PubMed Tutorial.

http://www.nlm.nih.gov/bsd/disted/pubmedtutorial/

References:
Haynes, R.B. (2007). Of studies, syntheses, synopses, summaries and systems: the “5S”
evolution of information services for evidence-based healthcare decisions. Evidence-
Based Nursing, 10, 6-7.
Part 4. Appraise. How can I decide if the particular study is good
enough to apply?
(1 hour)

Introduction
Unless you have found a pre-appraised article (such as a synopsis from an evidence-based
journal or summary in Health-Evidence.ca), you will have to assess the methods of the study.
This process is known as critical appraisal and what you are judging is the quality of the
methods, trying to answer the question:

Was this study done well enough that I can be confident in the findings?

There are key quality criteria for any types of studies that you find in the 5 S Pyramid (Part 3
above - systems, summaries, synopses, syntheses, and studies). It is not the intent of this
learning module to detail the critical appraisal process for every kind of research report, but to
make you aware of the resources where you can get the tools and explanations about their
application.

▲ A word of caution! Neophytes to critical appraisal sometimes throw out relatively well-
done studies from consideration because they are not perfect. There are no perfect studies. As
you become more familiar with the process, you will see that there are some criteria that relate
to larger concerns, and would therefore be ‘fatal flaws’ for which you would reject the study.
However, some other criteria are not so critical and, even if the study has not fulfilled that
particular criterion, you would still consider implementing the intervention.

Most of the available tools are built from the key criteria developed by the Evidence-Based
Medicine Working Group for the series published in JAMA, later collected in a book (Guyatt
& Rennie, 2002). The particular tools below are no exception, but are recommended because
they are more self-explanatory than most. With each of these recommended tools, you get an
explanation of the criterion they are asking you to rate – handy to have for neophyte critical
appraisers.

Once again, we are using our scenario, about preventing obesity in children with school-based
physical activity programs. Use the scenario for every section on appraising below.

1. Appraising a guideline.

You know about the ‘5S Pyramid”, so you started at the top and searched for a guideline,
which you found at the National Guideline Clearinghouse (www.guidelines.gov). How do you
decide if the guideline was well done?

An internationally accepted standard for critiquing guidelines is the AGREE tool, from the
AGREE (Appraisal of Guidelines Research and Evaluation) collaboration
(http://www.agreecollaboration.org). The tool is at
http://www.agreecollaboration.org/instrument/. While it looks intimidating simply due to its
number of pages, its length is due to the explanations given for every criterion and for the
scoring. It is very self-explanatory, and depending on the guideline, can usually be completed
in well under an hour.

The tool is set up to be scored by individuals, but can also be used in a consensus process,
where each rating is discussed. As an individual using the tool, you get a sense of where there
are problems that my affect your use of the guideline; you can decide if there is a ‘fatal flaw’
that would lead you to reject the guideline, or if there are minor flaws, such that you can still
get a very good protocol from a guideline. For example, one of the criteria is related to whether
the guideline group has set a time for the next review and update of the guideline. Even if "no",
this would not be a fatal flaw. Conversely, the lack of thorough search for relevant literature
on which to base the guideline would be a reason to reject the guideline.

Practice!

Practice rating a guideline:

a) Print the AGREE tool (http://www.agreecollaboration.org/instrument).
b) Go back to the Systems part of section 3 and search for a relevant guideline.
c) Try out the AGREE tool to rate the guideline you found in b) above.

2. Appraising a review (synthesis)

The Critical Appraisal Skills Program (CASP) of the Public Health Resources Unit in the UK
has a series of tools with explanations of criteria that are freely accessible on-line for personal
use (http://www.phru.nhs.uk/Pages/PHD/resources.htm). As with a guideline, you want to
know if you can really rely on the systematic review that you found regarding physical activity
in the schools to prevent obesity in children. The criteria ask you about factors such as the
clarity of the review question, if the search was thorough, the size of the effect and how precise
it was, and if the interventions described in the review could be applied to your population or
patients.
Practice!

Practice rating a systematic review:

a) Print the tool to appraise systematic reviews
http://www.phru.nhs.uk/learning/casp_s.review_tool.pdf
b) Go back to the synthesis part of section 3 and search for a relevant systematic review
or meta-analysis
c) Try out the CASP review tool to rate the systematic review you found in b) above.

3. Appraising a primary study about an intervention

The Critical Appraisal Skills Program (CASP) of the Public Health Resources Unit in the UK
has a series of tools with explanations of criteria, that are freely accessible on-line for personal
use (http://www.phru.nhs.uk/Pages/PHD/resources.htm). As with a systematic review, you
want to know if you can really rely on the individual studies you find about physical activity.
The criteria ask you about factors such as the size of the effect, the precision of the results, and
if the interventions described in the review could be applied to your population or patients.
Sometimes in health care research we have information about interventions through the highest
level of evidence, the randomized controlled trial; for other types of health care studies, we
only have evidence at the level of cohort studies or case-controls. Therefore, there are different
review criteria for different types of studies. Randomized trials have the greatest ability to
control for confounders or bias.

For practice rating a primary study:

a) Go back to the single studies part of section 3 and search for a relevant intervention
study, preferably a randomized trial.
b) Print the tool to appraise randomized trials
http://www.phru.nhs.uk/Doc_Links/rct%20appraisal%20tool.pdf
c) Try out the CASP tool to rate the randomized trial you found in a) above.
 Critical Appraisal Skills Programme (CASP) (UK):
http://www.phru.nhs.uk/Pages/PHD/resources.htm

Form to critically appraise systematic reviews:

http://www.phru.nhs.uk/Doc_Links/S.Reviews%20Appraisal%20Tool.pdf

Form to critically appraise randomized trials:

http://www.phru.nhs.uk/Doc_Links/rct%20appraisal%20tool.pdf

Form to critically appraise cohort studies:

http://www.phru.nhs.uk/Doc_Links/cohort%2012%20questions.pdf

Form to critically appraise case control studies:

http://www.phru.nhs.uk/Doc_Links/Case%20Control%2011%20Questions.pdf

Form to critically appraise economic evaluation studies:

http://www.phru.nhs.uk/Doc_Links/Economic%20Evaluations%2010%20Questi
ons.pdf

Form to critically appraise qualitative studies:

http://www.phru.nhs.uk/Doc_Links/Qualitative%20Appraisal%20Tool.pdf

OTHER RESOURCES

Shea B. J., Grimshaw J. M. , Wells G. A., Boers M., Anderson N., Hamel C., Porter A. C.,
Tugwell P., Moher D., & Bouter L.M. (2007). Development of AMSTAR: a
measurement tool to assess the methodological quality of systematic reviews. BMC
Medical Research Methodology, 1-7. This article is available from
http://www.biomedcentral.com/1471-2288/7/10

Duke University Medical Center. Introduction to Evidence-Based Medicine. Evaluating the

Evidence. http://www.hsl.unc.edu/services/tutorials/ebm/Evidence.htm
References:

GRADE Working Group (2004). Grading quality of evidence and stregth of recommendations.
British Medical Journal, 328,1490-7.

Guyatt, G. & Rennie, D. (Eds) (2002). Users’ Guides to the Medical Literature: A manual for
Evidence-Based Clinical Practice. American Medical Association.
Part 5. Integrate. How do I decide which of multiple studies to use?
(0.5 hours)

You do a focused search on the top levels of the 5S Pyramid and find

1 guideline

5 systematic reviews

26 individual studies
about the effectiveness of daily physical activity on obesity in children.

Should you read all this information?

Should you critically appraise it all?
What is the most efficient way to get through this information?

Have a look at Figure 5.1 to get a sense of how you would decide about use of the varying
levels of evidence you have found. Using the 5S Pyramid from Section 2, you would start with
the retrieval and reading of the guideline; if it is relevant, you would go on to apply the
AGREE tool (Section 4) to appraise the guideline. If you and your colleagues conclude that
this is a relevant guideline of sufficient quality that you can utilize it, and it is published in the
past two years, that may be the extent of your work. However, if you conclude it is not a good
review, you need to go to the systematic reviews.

One way to approach multiple systematic reviews of the same topic is to look first at those with
the most recent date of searching. The date of publication may not be relevant, but in the
methods under search strategy, it will detail the years included in the search. The critical
appraisal can then be ordered chronologically, starting with the reviews with the most recent
searches. That will give you some confidence in how up-to-date is the publication.

If you find that the systematic reviews are recent and of high quality, the only reason to search
for additional primary studies would be to take into consideration those published after the
search dates in the systematic reviews. Were their results similar to the results reported in the
systematic review or have the more recent studies presented different conclusions?

Finally, in order to get the details of more complex interventions (almost anything beyond drug
therapy), you would have to look up the primary studies that were included in the guideline
development or systematic review, as these syntheses cannot include enough detail in their
publication to allow the clinician to implement the intervention.
 1. Guidelines found?

yes no 2. Systematic review found?

Critically appraise: yes no

good enough to use?

Critically appraise:
good enough to use?
3. Single study found?

yes no
yes

yes
no

Critically appraise:
Implementation good enough to use?

4. Expert consensus

Figure 5.1 Decision Tree for Using Research Evidence

What if you find no research evidence at any level of the 5S Pyramid? Some health care
questions have not yet been answered by research. Expert opinion is the next best option.
Going back to the scenario, your specific question might be about school-based daily physical
activity of three hours or more versus one hour or less. That question has not been addressed
by research and may never be! If you are really intent on this question, you might get an expert
to estimate the calorie expenditure and they might guess which would utilize more energy.

Otherwise, if there are no published studies, you might be seeking more general information
from experts of colleagues, like:

What have people tried?

How did it seem to work?

Were patients/populations open to trying the intervention?

Were patients/populations satisfied with the results?
Part 6. Adapt. How do I use the information from #5 in decision-
making/policy brief?
(1 hour)

Research evidence comes from studies of particular samples of populations. You will have
noticed when looking at the critical appraisal tools that all include criteria related to “can I use
this research with my patients (or population)?” In every instance, you must consider the
inclusion/exclusion of participants in the study.

▲ Some people take this to the extreme and claim they can only use studies done in their
own institution. A better question might be to ask if there are important reasons why you
cannot apply this evidence to your situation.

For example, from our scenario, we may not be interested in studies of physical activities done
in boarding schools where the time for activity may have been done outside of the regular
curriculum. You know this cannot work without the ‘captive audience” as the students at your
school would mostly not be available, particularly if busing is used at your school.

In addition, you must consider how this research evidence fits with outer factors.
Consider:

Magnitude of health issue in local setting

Magnitude of the potential “reach” of the intervention

Costs (direct and indirect) of implementing the intervention

Availability of resources (personnel)

Organizational expertise and capacity

Political acceptability of the intervention

Social acceptability of the proposed intervention

A tool for assessing applicability in community interventions is available:

 Assessment of Applicability & Transferability
Construct Factors Questions to Ask
Applicability Political
(feasibility) acceptability or • Will the intervention be allowed or supported in
leverage current political climate?
• Will there be public relations benefit for local
government?
• Will this program enhance the stature of the
organization?
• Will the public and target groups accept and support
the intervention in its current format?

Social
acceptability • Will the target population be interested in the
intervention? Is it ethical?

Available
essential • Who/what is available/essential for the local
resources implementation?
(personnel and • Are they adequately trained? If not, is training
financial) available and affordable?
• What is needed to tailor the intervention locally?
• What are the full costs (supplies, systems, space
requirements for staff, training,
technology/administrative supports) per unit of
expected outcome?
• Are the incremental health benefits worth the costs
of the intervention?

Organizational
expertise and • Is the current strategic plan/operational plan in
capacity alignment with the intervention to be offered?
• Does this intervention fit with its mission and local
priorities?
• Does it conform to existing legislation or regulations
(either local or provincial?) Does it overlap with
existing programs or is it symbiotic?)
• Any organizational barriers/structural issues or
approval processes to be addressed?
• Is the organization motivated (learning
organization)?
 Transferability Magnitude of
(generalizability) health issue in • Does the need exist?
local setting • What is the baseline prevalence of the health issue
locally?
• What is the difference in prevalence of the health
issue (risk status) between study and local settings?

Magnitude of
the “reach” and • Will the intervention broadly “cover” the target
cost- population?
effectiveness
of the
intervention
Target
population • Are they comparable to the study population?
characteristics • Will any difference in characteristics (ethnicity,
socio-demographic variables, number of persons
affected) impact intervention effectiveness locally?

Buffett et al, 2007

http://www.nccmt.ca/pubs/2007_12_AT_tool_v_nov2007_ENG.pdf

Practice!

Once again, go back to the scenario. At this point, you are satisfied that the research that you
have found supports the inclusion of more active daily physical activity periods for school
children.
You are now developing that recommendation for a policy that will be presented at the local
school board. Using the community where you actually live, consider the factors in the
Applicability and Transferability Tool above. Where will the strengths/supports be and where
will the barriers/weaknesses be?

Develop an argument for why the evidence can or cannot be applied with your own
community.
References:

Buffett, C., Ciliska, D., & Thomas, H. (2007). Can I use this evidence in my program decision?
Assessing the applicability and transferability of evidence. National Collaborating
Centre for Methods and Tools.
http://www.nccmt.ca/pubs/2007_12_AT_tool_v_nov2007_ENG.pdf

Fervers B, Burgers JS, Haugh M et al (2006). Adaptation of clinical guidelines: A review of

methods and experiences. International Journal of Health Care, 18, 167-76.

Grimshaw J, Thomas RE, MacLennan G et al (2004). Effectiveness and efficiency of guideline

dissemination and implementation strategies. Health Technology Assessment, 8:iii-iv-
72.

Toman C, Harrison MB, & Logan J (2001). Clinical Practice Guidelines: Necessary but not
Sufficient for Evidence-based Patient Education and Counseling. Patient Education
and Counseling, 42(3), 279-87.
Part 7. Apply. How do I develop the implementation plan?
(0.5 hours)

Back to our scenario, you have now framed the question, searched for, found and appraised the
research evidence as good enough to use and you have concluded that the intervention of daily
physical activity in the schools is feasible (can be transferred) in your community.

Going back to Figure 1 in Part 1, we are now up to the left outer circle section about
“Assessing barriers to knowledge use”. Some of the barriers may have been identified as you
considered the applicability/transferability in Part 6, or in your consideration of general barriers
to Evidence-Informed Decision Making in Part 1.

Practice!

Make a list of the barriers you think would be active in instituting daily physical activity in the
schools (our scenario).
Barriers relate to:

The individuals who will use this information (e.g., What barriers will teachers have?
Students?)

The organizations (e.g., School boards? Curriculum?)

The broader culture (e.g., Prevailing attitudes)

The change itself (e.g., How much time will it take away from the usual curriculum?
How complex is the task? Will any school personnel require more training?)

Now use the same list and consider the supports that would be available in each of the
categories.

Supports relate to:

The individuals who will use this information (e.g., Teachers? Students?)

The organizations (e.g., Parents? Board of Health?)


The broader culture (e.g., Attitudes toward fitness)

The change itself (e.g., A welcome break from in-class activities?)

Together, this assessment of barriers and supports is sometimes called a situational analysis.
The analysis will help you plan for strategies to implement change. As with any planned
change, it is important to consider both how to build support as well as how to break down
barriers. Go back to Figure 1 in Part 1. We are now up to the left outer circle section about
“Select, tailor and implement interventions”.

John Lavis and colleagues (2003, 2004) proposed five key questions to assist in planning for
knowledge exchange. They are key questions to answer whether you are a researcher, manager,
practitioner or policy-developer.

What (is the message)? This translates or transforms research findings into an
actionable message.

To Whom (the audience)? Be specific when defining who will need to get the
message. Understand who are making the decisions.

By Whom (the messenger)? Is the messenger credible and is there a chance for the
audience to partner with the messengers?

How (transfer method)? What is the budget, preferred mechanism for learning new
information, is the audience actively engaged in selected the mechanism? Is the transfer
mechanism evidence-based? Is the intervention tailored to overcome the audience’s
identified barriers?

With what expected impact (evaluation)? What does the knowledge translation ,
synthesis and exchange (KTSE) project hope to change?

Practice!

Back to our scenario, we have found good evidence related to daily physical activity for
children done in the schools. Answering the following questions will go a long way to
helping you develop an action plan.
 • What (is the message)?

• To Whom (the audience)?

• By Whom (the messenger)?

• How (transfer method)?

OTHER RESOURCES

DiCenso, A., Virani, T., Bajnok, I., Borycki, E., Davies, B., Graham, I. et al. (2002). A toolkit
to facilitate the implementation of clinical practice guidelines in healthcare settings.
Hospital Quarterly, 5(3), 55-60.

Registered Nurses Association of Ontario. Toolkit: Implementation of Clinical Practice

Guidelines. http://www.rnao.org/Page.asp?PageID=924&ContentID=823

Effective Practice and Organization of Care. http://www.epoc.cochrane.org/en/index.html

This is a collaborative review group of the Cochrane Collaboration, focusing on change in
health care practitioner practice and how health care is organized.

References:

Bosch M, Van der Weijden T, Wensing M, & Grol R (2007). Tailoring quality improvement
interventions to identified barriers: a multiple case analysis. Journal of Evaluation in
Clinical Practice, 13, 161-168.

Dijkstra R, Wensing M, Thomas R, Akkermans R, Braspenning J, Grimshaw J, & Grol R

(2006). The relationship between organisational characteristics and the effects of
clinical guidelines on medical performance in hospitals, a meta-analysis. BMC Health
Services Research, 6, 53.

Grimshaw, J., Eccles, M., Thomas, R., MacLennan, G., Ramsay, C., Fraser, C., & Vale, L.,
(2006). Toward evidence-based quality improvement. Evidence (and its limitations) of
the effectiveness of guideline dissemination and implementation strategies 1966-1998.
Journal of General Internal Medicine, 21, S2, S14-20.

Grol R, & Grimshaw J (2003). From best evidence to best practice: effective implementation of
change in patients’ care. Lancet, 362, 1225-30
Lavis, J. N., Posada, F. B., Haines, A., & Osei, E. (2004). Use of research to inform public
policymaking. Lancet, 364, 1615-1621.

Lavis, J.N., Robertson, D., Woodside, J.M., McLeod, CB, Abelson,J. & the Knowledge
Transfer Study Group (2003). How can research organizations more effectively transfer
research knowledge to decision-makers? The Milbank Quarterly, 81(2), 221-248.

Wensing M, Van der Weijden T, & Grol R (1998). Implementing Guidelines and Innovations in
Primary Care: Which Interventions are Effective? British Journal of General Practice,
48, 991-997.
Part 8. How do I evaluate?
(0.5 hours)

For those who like logical progressions, here is a nice sequence:

Introduce policy ► change practitioner behavior ► impact population outcome

Sometimes it even works! A hospital EBP group reviewed the literature on skin preparation for
venipuncture and concluded the literature favored chlorhexidine. They created a hospital wide
policy, got their ordering/stores department to replace the alcohol swabs with chlorhexidine on
the venipuncture preparation trays, had some strategies to communicate these changes to staff,
and behold, the staff switched to using chlorhexidine for skin prep. A lovely example of the
linear sequence above!

But in the scenario, some school boards have had a policy of quality daily fitness for some
years already. Do the schools always follow it? No – they counter with “curriculum overload”
and lack of time to fit in the fitness schedule.

Where the school principals insist on fitting daily activity in the schedule, there are students
who take every opportunity to miss the event or participate with little enthusiasm and less
energy expenditure, with equal impact on their body composition! To the opposite effect, you
can see that one keen teacher, despite a lack of support by the principal, ensures that her
students get fifteen minutes of high level activity each day, even when she is supposedly
teaching math. Practitioners can, therefore, introduce changes in their own practice without a
policy in place. Likewise, some students will take it upon themselves to get more than the
suggested share of high level activity during recess breaks, so will get the fitness benefit
without policy or teacher intervention!

If the practitioners follow the policy, is there an impact on the population? In the case of this
scenario, can you prevent obesity? Five years later, is the rate of obesity at your school lower
than before the policy was introduced? Does the obesity level of the children change?

Every step in this sequence is a fruitful area for program evaluation or a more formal funded
research project. Studying how policy development and implementation take place, including
factors that affect each stage, would increase our understanding about the policy process. We
do not know very much about uptake of policy directives or how they get adapted at the local
level. From our scenario, how does the policy about physical activity get carried out by the
schools and how much does the principal alter the policy through interpretation, or how
carefully do they monitor if the teachers are following the policy, or how many ‘shortcuts’ do
teachers take, because they feel like they are getting behind in the curriculum?

There are some tools to evaluate research utilization in general. However, you can see that,
with the scenario, you would be wise to evaluate more specific processes and outcomes, such
as how many teachers did the daily activity, what percentage of days, estimated costs of
training teachers, students grades, student fitness activities and rate of obesity in students.
Practice!

How would you assess the outcome of your planned change (Part 7)?
What outcomes could you assess in relation to:

Policy development

Policy uptake

Teacher use of the policy

Student participation

Student outcomes

Resolution of the scenario

You now have a beginning understanding of evidence-informed decision making and are very
prepared for your participation in the group. You have framed a clear question about increasing
physical activity in the schools, and have considered what the barriers and supports might be
both for policy development and implementation. You also have some ideas about outcomes to
consider for evaluation in terms of reach, penetration, fidelity to the intervention and outcomes
for students. You are a valuable and well-informed group member.
References:

Eccles M, Grimshaw J, Campbell M, & Ramsay C (2003). Research designs for studies
evaluating the effectiveness of change and improvement strategies. Quality and Safety
in Health Care, 12(1):47-52.

Grimshaw JM, & Eccles MP (2004). Is evidence-based implementation of evidence-based care

possible? Medical Journal of Australia, 180(6 Suppl), S50-S51.

Grol R, & Grimshaw J (2003). From best evidence to best practice: effective implementation
of change in patients' care. The Lancet, 362(9391), 1225-30
Glossary
Bias: a systematic error or departure from the truth in results.

Cohort study: a group of people with a common set of characteristics or set of characteristics
that are followed up for a period of time to determine the incidence of an outcome; there is no
comparison group.
Cullum, N., Ciliska, D., Haynes, R.B., & Marks, S. (2008). Evidence-Based Nursing. An Introduction.
Oxford: Blackwell

Case-control study: an observational study that begins by comparing patients who have the
health problem (cases) and control participants who do not have the health problem, and then
looking back in time to identify the existence of possible causal factors. For example, the
identification of patients with and without lung cancer and looking back in time to determine
past smoking behavior (exposure to tobacco).
Dawson-Saunders, B., Trapp, R.G. (1994). Basic and Clinical Biostatistics. Norwalk: Appleton & Lange

Case report: detailed report of a person detailing signs, symptoms, progression of illness

Case series: a report on a series of patients with an outcome of interest. There is no

comparison group.

Confounder: a variable that affects the observed relationship between two other variables. For
example, alcohol consumption is related to lung cancer but does not cause the disease; instead,
both alcohol and lung cancer are related to smoking (the confounder), which causes lung
cancer.
Crombie, I.K. (1996). The pocket guide to critical appraisal: A handbook for Healthcare Professionals.
London: BMJ Publishing Group.

Evidence: information or facts that are systematically obtained (i.e., obtained in a manner that
is replicable, observable, credible, verifiable, or basically supportable).
Rycroft-Malone, J., & Stetler, C.B. (2004). Commentary on evidence, research, knowledge: a call for
conceptual clarity. Worldviews on Evidence-Based Nursing; 1(2):98-101.

Evidence-based medicine: the conscientious, explicit and judicious use of the current best
evidence in making decisions about the care of individual patients
Sackett. D.L., Rosenberg, W.M., Muir-Gray, J.A., Haynes, R.B., Richardson, W.S. (1996). Evidence-
based medicine: what it is and what it isn’t. BMJ, 328, 535-536

Evidence-informed decision-making: the use of evidence that contributes to decision making

about particular problems or issues about best use of resources within institutions and across
the healthcare system.
Canadian Health Services Research Foundation (2006). Weighing Up the Evidence. Making evidence-
informed guidance accurate, achievable, and acceptable. A summary of the workshop held on September
29, 2005. http://www.chsrf.ca/other_documents/evidence_e.php#definition, last downloaded May 2008).
Dissemination: involves identifying the appropriate audience and tailoring the message and
medium to the audience. Dissemination activities can include such things as
summaries/briefings to stakeholders, educational sessions with patients, practitioners and/or
policy makers, engaging knowledge users in developing and executing
dissemination/implementation plan, tools creation, and media engagement.
Canadian Institutes for Health Research About Knolwedge Translation.
http://www.cihr-irsc.gc.ca/e/29418.html

Knowledge translation: is a dynamic and iterative process that includes synthesis,

dissemination, exchange and ethically-sound application of knowledge to improve the health of
Canadians, provide more effective health services and products and strengthen the health care
system. This process takes place within a complex system of interactions between researches
and knowledge users which may vary in intensity, complexity and level of engagement
depending on the nature of the research and the findings as well as the needs of the particular
knowledge user.
Canadian Institutes for Health Research About Knolwedge Translation.
http://www.cihr-irsc.gc.ca/e/29418.html

Knowledge exchange (formerly knowledge transfer): is collaborative problem-solving

between researchers and decision makers that happens through linkage and exchange. Effective
knowledge exchange involves interaction between decision makers and researchers and results
in mutual learning through the process of planning, producing, disseminating, and applying
existing or new research in decision-making.
Canadian Health Services Research Foundation . Glossary of Knowledge Exchange Terms Used by the
Foundation. http://www.chsrf.ca/keys/glossary_e.php

Meta-analysis: a method for combining the results of several independent studies that measure
the same outcomes so that an overall summary statistic can be calculated.
Dawson-Saunders, B., Trapp, R.G. (1994). Basic and Clinical Biostatistics. Norwalk: Appleton & Lange.

Phenomenology: an approach to enquiry that emphasizes the complexity of human experience

and the need to understand that experience holistically, as it is actually lived.
Polit, D.F., & Hungler, B.P. (1997). Essentials of Nursing Research: Methods, Appraisal and
Utilization. Philadelphia: Lippincott.

Qualitative research: research that aims to generate an understanding of complex,

unquantifiable phenomena, such as people’s experiences or perceptions. A few examples of
types of qualitative research include phenomenology, grounded theory, participatory action
research, and ethnography.
Cullum, N., Ciliska, D., Haynes, R.B., & Marks, S. (2008). Evidence-Based Nursing. An Introduction.
Oxford: Blackwell.

Randomized controlled trial (RCT): a study design in which individuals are randomly
allocated to receive alternative preventive, therapeutic or diagnostic interventions and then
followed up to determine the effect of the interventions (one of the alternatives might be no
intervention).
 Cullum, N., Ciliska, D., Haynes, R.B., & Marks, S. (2008). Evidence-Based Nursing. An Introduction.
Oxford: Blackwell.

Synthesis: the contextualization and integration of research findings of individual research

studies within the larger body of knowledge on the topic. A synthesis must be reproducible and
transparent in its methods, using quantitative and/or qualitative methods. It could take the form
of a systematic review, follow the methods developed by the Cochrane Collaboration, result
from a consensus conference or expert panel and may synthesize qualitative or quantitative
results. Realist syntheses, narrative syntheses, meta-analyses, meta-syntheses and practice
guidelines are all forms of synthesis.
Canadian Institutes for Health Research. About Knolwedge Translation.
http://www.cihr-irsc.gc.ca/e/29418.html

Systematic review: a research summary of all evidence that relates to a particular question; the
question could be one of intervention effectiveness, causation, diagnosis or prognosis. The
systematic review process follows a rigorous methodology for searching, retrieval, relevance
and quality rating, data extraction, data synthesis and interpretation.
Cullum, N., Ciliska, D., Haynes, R.B., & Marks, S. (2008). Evidence-Based Nursing. An Introduction.
Oxford: Blackwell