DOI: 10.7860/JCDR/2017/26047.

9942 Review Article

Critical Appraisal of
Education Section

Clinical Research

Azzam Al-Jundi1, Salah Sakka2

Abstract
Evidence-based practice is the integration of individual clinical expertise with the best available external clinical evidence from
systematic research and patient’s values and expectations into the decision making process for patient care. It is a fundamental
skill to be able to identify and appraise the best available evidence in order to integrate it with your own clinical experience and
patients values. The aim of this article is to provide a robust and simple process for assessing the credibility of articles and their
value to your clinical practice.

Keywords: Evidence-based practice, Method assessment, Research design

INTRODUCTION include authors’ declarations of interest and potential market bias.

Decisions related to patient value and care is carefully made following Attention should be paid to any declared funding or the issue of a
an essential process of integration of the best existing evidence, research grant, in order to check for a conflict of interest [2].
clinical experience and patient preference. Critical appraisal is II. ABSTRACT: Reading the abstract is a quick way of getting
the course of action for watchfully and systematically examining to know the article and its purpose, major procedures and
research to assess its reliability, value and relevance in order to methods, main findings, and conclusions.
direct professionals in their vital clinical decision making [1]. a. Aim of the study: It should be well and clearly written.
Critical appraisal is essential to: b. Materials and Methods: The study design and type of groups,
• Combat information overload; type of randomization process, sample size, gender, age,
• Identify papers that are clinically relevant; and procedure rendered to each group and measuring tool(s)
should be evidently mentioned.
• Continuing Professional Development (CPD).
c. Results: The measured variables with their statistical analysis
Carrying out Critical Appraisal:
and significance.
Assessing the research methods used in the study is a prime step in
d. Conclusion: It must clearly answer the question of interest.
its critical appraisal. This is done using checklists which are specific
to the study design. III. Introduction/ Background section:

Standard Common Questions: An excellent introduction will thoroughly include references to

earlier work related to the area under discussion and express the
1. What is the research question?
importance and limitations of what is previously acknowledged [2].
2. What is the study type (design)?
-Why this study is considered necessary? What is the purpose of
3. Selection issues. this study? Was the purpose identified before the study or a chance
4. What are the outcome factors and how are they measured? result revealed as part of ‘data searching?’
5. What are the study factors and how are they measured? -What has been already achieved and how does this study be at
6. What important potential confounders are considered? variance?

7. What is the statistical method used in the study? -Does the scientific approach outline the advantages along
with possible drawbacks associated with the intervention or
8. Statistical results. observations?
9. What conclusions did the authors reach about the research IV. Methods and Materials section: Full details on how the
question? study was actually carried out should be mentioned. Precise
10. Are ethical issues considered?? information is given on the study design, the population, the
The Critical Appraisal starts by double checking the following sample size and the interventions presented. All measurements
main sections: approaches should be clearly stated [3].
I. Overview of the paper: V. Results section: This section should clearly reveal what
actually occur to the subjects. The results might contain raw
a. The publishing journal and the year
data and explain the statistical analysis. These can be shown
b. The article title: Does it state key trial objectives? in related tables, diagrams and graphs.
c. The author (s) and their institution (s) VI. Discussion section: This section should include an absolute
The presence of a peer review process in journal acceptance comparison of what is already identified in the topic of interest
protocols also adds robustness to the assessment criteria for and the clinical relevance of what has been newly established.
research papers and hence would indicate a reduced likelihood of A discussion on a possible related limitations and necessitation
publication of poor quality research. Other areas to consider may for further studies should also be indicated.

Journal of Clinical and Diagnostic Research. 2017 May, Vol-11(5): JE01-JE05 1

 Azzam Al-Jundi and Salah Sakka, Critical Appraisal of Clinical Research www.jcdr.net

Clinical Relevance and Suggested 2- What is the study type (design)?

Clinical Questions
Best Method of Investigation The study design of the research is fundamental to the usefulness
Aetiology/Causation What caused the disorder and how is this related to the of the study.
development of illness.
Example: randomized controlled trial - case-control study- In a clinical paper the methodology employed to generate the
cohort study. results is fully explained. In general, all questions about the related
Therapy Which treatments do more good than harm compared with clinical query, the study design, the subjects and the correlated
an alternative treatment?
Example: randomized control trial, systematic review, meta-
measures to reduce bias and confounding should be adequately
analysis. and thoroughly explored and answered.
Prognosis What is the likely course of a patient’s illness? Participants/Sample Population:
What is the balance of the risks and benefits of a treatment?
Example: cohort study, longitudinal survey. Researchers identify the target population they are interested in. A
Diagnosis How valid and reliable is a diagnostic test? sample population is therefore taken and results from this sample
What does the test tell the doctor? are then generalized to the target population.
Example: cohort study, case –control study
The sample should be representative of the target population from
Cost- effectiveness Which intervention is worth prescribing?
which it came. Knowing the baseline characteristics of the sample
Is a newer treatment X worth prescribing compared with
older treatment Y? population is important because this allows researchers to see how
Example: economic analysis closely the subjects match their own patients [4].
[Table/Fig-1]: Categories of clinical questions and the related study designs. Sample size calculation (Power calculation): A trial should be large
Does it summarize the main findings of the study and relate them to enough to have a high chance of detecting a worthwhile effect
any deficiencies in the study design or problems in the conduct of if it exists. Statisticians can work out before the trial begins how
the study? (This is called intention to treat analysis). large the sample size should be in order to have a good chance
of detecting a true difference between the intervention and control
• Does it address any source of potential bias?
groups [5].
• Are interpretations consistent with the results?
• Is the sample defined? Human, Animals (type); what population
• How are null findings interpreted? does it represent?
• Does it mention how do the findings of this study relate to • Does it mention eligibility criteria with reasons?
previous work in the area?
• Does it mention where and how the sample were recruited,
• Can they be generalized (external validity)? selected and assessed?
• Does it mention their clinical implications/applicability? • Does it mention where was the study carried out?
• What are the results/outcomes/findings applicable to and will • Is the sample size justified? Rightly calculated? Is it adequate to
they affect a clinical practice? detect statistical and clinical significant results?
• Does the conclusion answer the study question? • Does it mention a suitable study design /type?
- Is the conclusion convincing? • Is the study type appropriate to the research question?
- Does the paper indicate ethics approval? • Is the study adequately controlled? Does it mention type of
- Can you identify potential ethical issues? randomization process? Does it mention the presence of
- Do the results apply to the population in which you are control group or explain lack of it?
interested? • Are the samples similar at baseline? Is sample attrition
- Will you use the results of the study? mentioned?
Once you have answered the preliminary and key questions and • All studies report the number of participants/specimens at the
identified the research method used, you can incorporate specific start of a study, together with details of how many of them
questions related to each method into your appraisal process or completed the study and reasons for incomplete follow up if
checklist. there is any.
1-What is the research question? • Does it mention who was blinded? Are the assessors and
participants blind to the interventions received?
For a study to gain value, it should address a significant problem within
the healthcare and provide new or meaningful results. Useful structure • Is it mentioned how was the data analysed?
for assessing the problem addressed in the article is the Problem • Are any measurements taken likely to be valid?
Intervention Comparison Outcome (PICO) method [3]. Researchers use measuring techniques and instruments that have
P = Patient or problem: Patient/Problem/Population: been shown to be valid and reliable.
It involves identifying if the research has a focused question. What is Validity refers to the extent to which a test measures what it is
the chief complaint? supposed to measure.
E.g.,: Disease status, previous ailments, current medications etc., (the extent to which the value obtained represents the object of
interest.)
I = Intervention: Appropriately and clearly stated management
strategy e.g.,: new diagnostic test, treatment, adjunctive therapy etc., - Soundness, effectiveness of the measuring instrument;
C= Comparison: A suitable control or alternative - What does the test measure?
E.g.,: specific and limited to one alternative choice. - Does it measure, what it is supposed to be measured?
O= Outcomes: The desired results or patient related consequences - How well, how accurately does it measure?
have to be identified. e.g.,: eliminating symptoms, improving function, Reliability: In research, the term reliability means “repeatability”
esthetics etc., or “consistency”
The clinical question determines which study designs are appropriate. Reliability refers to how consistent a test is on repeated
There are five broad categories of clinical questions, as shown in [Table/ measurements. It is important especially if assessments are made
Fig-1]. on different occasions and or by different examiners. Studies should

2 Journal of Clinical and Diagnostic Research. 2017 May, Vol-11(5): JE01-JE05

www.jcdr.net Azzam Al-Jundi and Salah Sakka, Critical Appraisal of Clinical Research

state the method for assessing the reliability of any measurements Data Analysis and Results:
taken and what the intra –examiner reliability was [6]. • Assessment of the statistical significance should be evaluated:
3- Selection issues: - Were the tests appropriate for the data?
The following questions should be raised: - Are confidence intervals or p-values given?
- How were subjects chosen or recruited? If not random, are • How strong is the association between intervention and
they representative of the population? outcome?
- Types of Blinding (Masking) Single, Double, Triple? • How precise is the estimate of the risk?
- Is there a control group? How was it chosen? • Does it clearly mention the main finding(s) and does the data
- How are patients followed up? Who are the dropouts? Why support them?
and how many are there? • Does it mention the clinical significance of the result?
- Are the independent (predictor) and dependent (outcome) • Is adverse event or lack of it mentioned?
variables in the study clearly identified, defined, and
• Are all relevant outcomes assessed?
measured?
• Was the sample size adequate to detect a clinically/socially
- Is there a statement about sample size issues or statistical
significant result?
power (especially important in negative studies)?
• Are the results presented in a way to help in health policy
- If a multicenter study, what quality assurance measures were
decisions?
employed to obtain consistency across sites?
• Is there measurement error?
- Are there selection biases?
• Is measurement error an important source of bias?
• In a case-control study, if exercise habits to be compared:
Confounding Factors:
- Are the controls appropriate?
A confounder has a triangular relationship with both the exposure
- Were records of cases and controls reviewed blindly?
and the outcome. However, it is not on the causal pathway. It makes
- How were possible selection biases controlled (Prevalence it appear as if there is a direct relationship between the exposure
bias, Admission Rate bias, Volunteer bias, Recall bias, Lead and the outcome or it might even mask an association that would
Time bias, Detection bias, etc.,)? otherwise have been present [9].
• Cross Sectional Studies: 6- What important potential confounders are considered?
- Was the sample selected in an appropriate manner (random, - Are potential confounders examined and controlled for?
convenience, etc.,)? - Is confounding an important source of bias?
- Were efforts made to ensure a good response rate or to 7- What is the statistical method in the study?
minimize the occurrence of missing data?
- Are the statistical methods described appropriate to compare
- Were reliability (reproducibility) and validity reported? participants for primary and secondary outcomes?
• In an intervention study, how were subjects recruited and - Are statistical methods specified insufficient detail (If I had
assigned to groups? access to the raw data, could I reproduce the analysis)?
• In a cohort study, how many reached final follow-up? - Were the tests appropriate for the data?
- Are the subject’s representatives of the population to which the - Are confidence intervals or p-values given?
findings are applied?
- Are results presented as absolute risk reduction as well as
- Is there evidence of volunteer bias? Was there adequate follow- relative risk reduction?
up time?
Interpretation of p-value:
- What was the drop-out rate?
The p-value refers to the probability that any particular outcome
- Any shortcoming in the methodology can lead to results that would have arisen by chance. A p-value of less than 1 in 20 (p<0.05)
do not reflect the truth. If clinical practice is changed on the is statistically significant.
basis of these results, patients could be harmed.
• When p-value is less than significance level, which is usually
Researchers employ a variety of techniques to make the methodology 0.05, we often reject the null hypothesis and the result is
more robust, such as matching, restriction, randomization, and considered to be statistically significant. Conversely, when
blinding [7]. p-value is greater than 0.05, we conclude that the result is not
Bias is the term used to describe an error at any stage of the study statistically significant and the null hypothesis is accepted.
that was not due to chance. Bias leads to results in which there are Confidence interval:
a systematic deviation from the truth. As bias cannot be measured,
Multiple repetition of the same trial would not yield the exact same
researchers need to rely on good research design to minimize bias
results every time. However, on average the results would be within
[8]. To minimize any bias within a study the sample population
a certain range. A 95% confidence interval means that there is a
should be representative of the population. It is also imperative to
95% chance that the true size of effect will lie within this range.
consider the sample size in the study and identify if the study is
adequately powered to produce statistically significant results, i.e., 8- Statistical results:
p-values quoted are <0.05 [9]. - Do statistical tests answer the research question?
4- What are the outcome factors and how are they measured? Are statistical tests performed and comparisons made (data
- Are all relevant outcomes assessed? searching)?
- Is measurement error an important source of bias? Correct statistical analysis of results is crucial to the reliability of
the conclusions drawn from the research paper. Depending on the
5- What are the study factors and how are they measured?
study design and sample selection method employed, observational
- Are all the relevant study factors included in the study? or inferential statistical analysis may be carried out on the results of
- Have the factors been measured using appropriate tools? the study.

Journal of Clinical and Diagnostic Research. 2017 May, Vol-11(5): JE01-JE05 3

 Azzam Al-Jundi and Salah Sakka, Critical Appraisal of Clinical Research www.jcdr.net

It is important to identify if this is appropriate for the study [9]. • Quality-control of studies included (type of study; scoring
- Was the sample size adequate to detect a clinically/socially system used to rate studies; analysis performed by at least
significant result? two experts).
- Are the results presented in a way to help in health policy • Homogeneity of studies.
decisions? • Presentation of results (clear, precise).
Clinical significance: • Applicability to local population.
Statistical significance as shown by p-value is not the same [Table/Fig-3] summarizes the guidelines for Preferred Reporting
as clinical significance. Statistical significance judges whether Items for Systematic reviews and Meta-Analyses PRISMA [13].
treatment effects are explicable as chance findings, whereas clinical
significance assesses whether treatment effects are worthwhile in
real life. Small improvements that are statistically significant might Title and abstract
Identification as a RCT in the title- Structured summary
not result in any meaningful improvement clinically. The following (trial design, methods, results, and conclusions)

questions should always be on mind: Introduction - Scientific background

- Objectives
- If the results are statistically significant, do they also have
Methods -Description of trial design and important changes to
clinical significance? methods
- If the results are not statistically significant, was the sample size -Eligibility criteria for participants
-The interventions for each group
sufficiently large to detect a meaningful difference or effect? -Completely defined and assessed primary and secondary
outcome measures
9- What conclusions did the authors reach about the study
-How sample size was determined
question? -Method used to generate the random allocation sequence
-Mechanism used to implement the random allocation
Conclusions should ensure that recommendations stated are sequence
suitable for the results attained within the capacity of the study. The -Blinding details
authors should also concentrate on the limitations in the study and -Statistical methods used

their effects on the outcomes and the proposed suggestions for Results -Numbers of participants, losses and exclusions after
randomization
future studies [10].
-Results for each group and the estimated effect size and its
- Are the questions posed in the study adequately addressed? precision (such as 95% confidence interval)
-Results of any other subgroup analyses performed
- Are the conclusions justified by the data?
Discussion -Trial limitations
- Do the authors extrapolate beyond the data? -Generalisability

- Are shortcomings of the study addressed and constructive Other information - Registration number
suggestions given for future research? [Table/Fig-2]: Summary of the CONSORT guidelines.

- Is the conclusion convincing?

Identification of the report as a systematic review,
- Bibliography/References: Title
meta-analysis, or both.
Do the citations follow one of the Council of Biological Editors’ (CBE) Abstract Structured Summary: background; objectives; eligibility
standard formats? criteria; results; limitations; conclusions; systematic review
registration number.
10- Are ethical issues considered?
Introduction - Description of the rationale for the review
If a study involves human subjects, human tissues, or animals, was - Provision of a defined statement of questions being
approval from appropriate institutional or governmental entities concentrated on with regard to participants, interventions,
comparisons, outcomes, and study design (PICOS).
obtained? [10,11].
Methods - Specification of study eligibility criteria
- Does the paper indicate ethics approval? - Description of all information sources
- Presentation of full electronic search strategy
- Can you identify potential ethical issues? - State the process for selecting studies
Critical appraisal of RCTs: Factors to look for: - Description of the method of data extraction from reports
and methods used for assessing risk of bias of individual
• Allocation (randomization, stratification, confounders). studies in addition to methods of handling data and
combining results of studies.
• Blinding.
Results Provision of full details of:
• Follow up of participants (intention to treat). - Study selection.
- Study characteristics (e.g., study size, PICOS, follow-up
• Data collection (bias). period)
• Sample size (power calculation). - Risk of bias within studies.
- Results of each meta-analysis done, including confidence
• Presentation of results (clear, precise). intervals and measures of consistency.
- Methods of additional analyses (e.g., sensitivity or subgroup
• Applicability to local population. analyses, meta-regression).
[Table/Fig-2] summarizes the guidelines for Consolidated Standards Discussion -Summary of the main findings including the strength of
of Reporting Trials CONSORT [12]. evidence for each main outcome.
- Discussion of limitations at study and outcome level.
Critical appraisal of systematic reviews: provide an overview of all - Provision of a general concluded interpretation of the
results in the context of other evidence.
primary studies on a topic and try to obtain an overall picture of the
results. Funding Source and role of funders.

In a systematic review, all the primary studies identified are critically [Table/Fig-3]: Summary of PRISMA guidelines.

appraised and only the best ones are selected. A meta-analysis

(i.e., a statistical analysis) of the results from selected studies may CONCLUSION
be included. Factors to look for: Critical appraisal is a fundamental skill in modern practice for
• Literature search (did it include published and unpublished assessing the value of clinical researches and providing an indication
materials as well as non-English language studies? Was of their relevance to the profession. It is a skills-set developed
personal contact with experts sought?). throughout a professional career that facilitates this and, through

4 Journal of Clinical and Diagnostic Research. 2017 May, Vol-11(5): JE01-JE05

www.jcdr.net Azzam Al-Jundi and Salah Sakka, Critical Appraisal of Clinical Research

integration with clinical experience and patient preference, permits [6] Sakka S, Al-ani Z, Kasioumis T, Worthington H, Coulthard P. Inter-examiner and
intra-examiner reliability of the measurement of marginal bone loss around oral
the practice of evidence based medicine and dentistry. By following
implants. Implant Dent. 2005;14(4):386-88.
a systematic approach, such evidence can be considered and [7] Rosenberg W, Donald A. Evidence based medicine: an approach to clinical
applied to clinical practice. problem-solving. BMJ. 1995;310(6987):1122-26.
[8] Stewart LA, Parmar, MK. Bias in the analysis and reporting of randomized
controlled trials. Int J Technol Assess Health Care. 1996;12(2):264-75.
REFERENCES [9] Egger M, Smith GD. Bias in location and selection of studies. BMJ.
[1] Burls, A. What is critical appraisal? 2016: London, Hayward Medical 1998;316(7124):61-66.
Communications. Available from: http://www.whatisseries.co.uk/what-is-critical- [10] Haynes RB. Of studies, syntheses, synopses, summaries, and systems: the "5S"
appraisal/ . evolution of information services for evidence-based healthcare decisions. Evid
[2] MacInnes A, Lamont T. Critical appraisal of a research paper. Scott Uni Med J. Based Med. 2006;11(6):162-64
2014;3 (1):10-17. [11] Al-Jundi A, Sakka S. Protocol writing in clinical research. J Clin Diagn Res.
[3] Richards D, Lawrence A. Evidence-based dentistry. Br Dent J. 1995;179(7):270- 2016;10(11):ZE10-ZE13
73. [12] Moher D, Hopewell S, Schulz KF, Montori V, Gøtzsche PC, Devereaux PJ, et al.
[4] Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS. Evidence CONSORT 2010 Explanation and Elaboration: updated guidelines for reporting
based medicine: what it is and what it isn’t. BMJ. 1996;312(7023):71-72. parallel group randomised trials. BMJ 2010;340:c869.
[5] Greenhalgh, T. How to read a paper: The basics of evidence based medicine. 5th [13] Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group. preferred reporting
ed, New York, United States; John Wiley & Sons: 2014. items for systematic reviews and meta-analyses: The PRISMA Statement. PLoS
Med. 2009;6(7):e1000097.

PARTICULARS OF CONTRIBUTORS:
1. Professor, Department of Orthodontics, King Saud bin Abdul Aziz University for Health Sciences-College of Dentistry, Riyadh, Kingdom of Saudi Arabia.
2. Associate Professor, Department of Oral and Maxillofacial Surgery, Al Farabi Dental College, Riyadh, KSA.

NAME, ADDRESS, E-MAIL ID OF THE CORRESPONDING AUTHOR:

Dr. Salah Sakka,
Associate Professor, Al-Farabi Dental College, Riyadh-11691, Kingdom of Saudi Arabia. Date of Submission: Dec 16, 2016
E-mail: [email protected] Date of Peer Review: Feb 14, 2017
Date of Acceptance: Apr 04, 2017
Financial OR OTHER COMPETING INTERESTS: None. Date of Publishing: May 01, 2017

Journal of Clinical and Diagnostic Research. 2017 May, Vol-11(5): JE01-JE05 5