SPECIAL ARTICLE

Critical Appraisal on Journal of Clinical Trials

Murdani Abdullah, M. Adi Firmansyah

Department of Internal Medicine, Faculty of Medicine, University of Indonesia - Cipto Mangunkusumo
Hospital. Jl. Diponegoro no. 71, Jakarta Pusat 10430, Indonesia. Correspondence mail:
[email protected]; [email protected].

ABSTRAK
Critical appraisal merupakan bagian dari kedokteran berbasis bukti (evidence-based medicine) diartikan
sebagai suatu proses evaluasi secara cermat dan sistematis suatu artikel penelitian untuk menentukan
reabilitas, validitas, dan kegunaannya dalam praktik klinis. Komponen utama yang dinilai dalam critical
appraisal adalah validity, importancy, dan applicability. Tingkat kepercayaan hasil suatu penelitian sangat
bergantung dari disain penelitian dimana uji klinis menempati urutan tertinggi. Telaah kritis meliputi semua
komponen dari suatu penelitian dimulai dari komponen pendahuluan, metodologi, hasil dan diskusi. Masing-
masing komponen memiliki kepentingan yang sama besarnya dalam menentukan apakah hasil penelitian
tersebut layak atau tidak digunakan sebagai referensi. Kemaknaan secara statistik yang didapat hendaknya
juga dibandingkan dengan kemaknaan secara klinis.

Kata kunci: critical appraisal, telaah kritis, evidence-based medicine, uji klinis, clinical trial.

ABSTRACT
Critical appraisal is an element of evidence-based medicine, which is defined as the process of carefully
and systematically examining a research article to determine its reliability, validity and value in clinical
practice. The major components evaluated in critical appraisal are validity, importancy, and applicability. The
level of reliability of study results depend on the design of study, in which clinical trial has the highest rank.
Critical appraisal includes all components of the research starting from the introduction, method, results and
discussion. Each component has similar value to establish whether the results can realistically be applied as a
referrence.
The results of statistical significance should also been compared with its clinical significance.

Key words: critical appraisal, evidence-based medicine, clinical trial.

INTRODUCTION practice. As clinicians, we must be able to

Nowadays, there has been increasingly appraise the information, particularly about
prevalent medical practice of evidence-based the clarity, accuracy, precision, reliability,
medicine (EBM) or evidence based practice. relevance and other supporting evidences
Principally, it is an approach trying to use including argumentation for making
current best scientific evidences obtained conclusion, the depth, breadth, and
from scientific research in making clinical considering fairness.
decision about treatment. The rate of reliability of study results of a
Certainly, not all information of scientific research extremely depends on study design,
journal can become reliable knowledge with which is known as the level of evidence. The
confirmed truth and subsequently suitable for hierarchy of study design which has high
being established into guidelines of clinical evidence to low evidence are as follows: [1]

337

Acta Medica Indonesiana - The Indonesian Journal of Internal Medicine

Murdani Abdullah Acta Med Indones-Indones J Intern
Med

randomized clinical trials (RCT), systematic what action to take from the findings including
reviews on RCT; [2] cohort study; [3] applying the eveidences for clinical practice; and
casecontrolled study; [4] case report or case 5) evaluate the practice performed according to
series; and [5] expert opinions.1 the evidencebased medicine (Figure 1).2,3
Method of appraising information should be Formulating question by using PICO
done sistematically to provide a good conclusion, concept can also be the element of critical
which is the best kind of information. It includes appraisal itself. The following is the example
appraisal of various sources of information as of PICO concept: “In a 50-year-old female
well as appraising the conclusion by providing patient who is diagnosed with type 2 diabetes
supporting evidences. One of the methods is mellitus (Patient), how great is the role of
critical appraisal. By performing critical
three-month diet and exercise program
appraisal, we are expected to be able to evaluate
results, validity and value of scientific article (Intervention) to help normalize the fasting
publications systematically. This manuscript is blood glucose (Outcome) compared to diet
going to discuss about critical appraisal and focus restriction alone (Comparator)?”
on the appraisal of clinical trial articles in Starting Critical Appraisal of Clinical Trial
scientific journals. Before we are performing critical appraisal
of a research article in a scientific journal, we
CRITICAL APPRAISAL should first comprehend the article. A good
comprehension of an article will make our
Definition evaluation better. In general, the basic
By definition, critical appraisal is regarded as components of a research article are
a process of evaluating a research article carefully introduction, methods, results and discussion
and systematically to determine the reliability, (Figure 2).
validity and application in clinical practice.2 In Usually, when we take a look at a research
other words, through critical appraisal, we decide article, the abstract is the first part that we read.
a research article is reliable or not. The ability to Most of us believe that the answer to our
perform critical appraisal should be one of basic questions is there in a concise form, while it
competence of a clinician in order to recognize would be too risky to consider an abstract as a
and use reliable study data efficiently. true summary of a research. Use it only to
The next question is how do we know that the decide about the topic that we are interested in
and to avoid further reading if by reading it we
obtained data has good quality and reliable? How
can already see a poor method of study.
could we decide which study is more reliable
In fact, we do not have to read the article
when we have data of two different conclusion on
word by word from beginning to end. In critical
the same topic? For this reason, therefore, we
appraisal, we would evaluate the article
should performm critical appraisal.
systematically. The list of questions that we
Critical appraisal is an important element of
should find the answer when reading a research
EBM. As we have known that there are five steps
in EBM when evaluating a clinical case, i.e.: 1) article are shown in Table 1.
formulating questions with the concept of PICO Overall, components of critical appraisal
(Population, Intervention, Comparation, for a research article are validity, importance
Outcome); 2) search for the evidence or and applicability, which is more familiar as the
reference; 3) appraise the evidence on the abbreviation of VIA as shown in Figure 3 as
component of validity and importancy; 4) decide the following.

Asking
Clinical problems Search Appraise Decide Evaluate
questions

PICO: Finding Critical

Population, Intervention, the evidence appraisal
Comparation, Outcome

Figure 1. Steps of evidence-based medicine and the role of critical appraisal Figure 2. Components of a
research article that should be comprehended before doing critical appraisal

338
Vol 44 • Number 4 • October 2012 Critical appraisal on journal of
clinical trials

Comprehend Evaluate

Outline Abstract

Introduction Degrees of problem, elaboration,

research question and hypothesis

Population, subjects, sample size,

Methods sampling, measurement, analysis plan

Research flow chart, subject’s characteristics,

Results baseline data, main outcome

Non-causal internal validity, causal internal validity,

Discussion external validity,importancy, applicability

Table 1. Critical appraisal checklist5

I. Study validity
Research question
- Is the research question well-defined that can be answered using this study design?
Randomization
- Were the patients randomized to the intervention and control groups by a well-defined method of randomization?
- Was the randomization list concealed from patients, clinicians and researchers?
- Do the patients in each group have similar characteristics at the beginning of the study?
Blinding
- Were the patients and clinicians kept blinded (masked) to which treatment was being given?
- Were they kept blinded until the end of the study?
Follow-up
- Were all patients counted at the end of the study?
- If not, how many patients were lost to follow up and for what reason?
- Were the patients analysed in the group they originally were randomized to?
Interventions and co-interventions
- Were the performed interventions described in sufficient detail to be followed by others?
- Other than intervention, were th two groups cared for in similar way of treatment?
II. Results
Selection of outcomes
- Does the article report all relevant outcomes including side effect?
Effect size
- Was there a difference between the outcomes of the treatments, and how big was the difference?
- How reliable is the results: what are the confidence intervals?
III. Applicability
Using results in your own setting
- Are your patient so different from those studied that the results may not apply to them?
- Is your environment so different from the one in the study that the methods could not be use there?

339
Murdani Abdullah Acta Med Indones-Indones J Intern
Med

validity of selection
Non-causal validity of information
internal validity validity of control for confounding
validity of analysis

Temporality
Degree of association
Validity Causal internal Dose response
validity Consistency
Coherency
Specificity
Biological plaussability

V External validity External validity 1

External validity 2
I
Importancy Clinical importance
A of the study results

Transportability,patient’s condition,
Applicability service capacity, economy,
socio-culture, religion

4 ,5
Figure 3. Components of critical appraisal Table 2. List of questions for the introduction of clinical
journal article4
Component Explanation

The component of validity is aimed to proof What is the data that Prevalence, incidence, impact.
whether the result is a fact or not; while the has been
presented to show
component of importancy and applicability are
the magnitude of
directed to evaluate the value and feasibility. problems?
The Introduction What data that Substantial and methodological
In the introduction component of a research has been known elaboration
article, there are usually two or three paragraph before
(elaborated)?
in brief statement. In this section, the researcher
tries to explain about the reason of why he/she Is there still any gap? Substantial and methodological
conducted the study. Generally, when we read it gap and confirmation of previous
study results.
carefully, there are a pattern of introduction
component in a research article. The pattern What are the major The most important objective of
includes the formulation of data to determine the aims (primary the study. It provides the basis
endpoint)? of sample size calculation and
degrees of problem (for instance the data of the study is usually designed to
prevalence, incidence, number of cases, the answer the major aims.
magnitude of effect in certain cases), elaboration
What are the minor It does not serve as the basis of
of data that has been known before, formulation aims (secondary sample size calculation. It may
of current gap (problems) and the aim or endpoint)? be elaborative and provides
hypothesis of study. The list of component baseline data for further studies.
associated with the introduction is shown on the What is the study It consists of non-inferiority trial
following Table 2. hypothesis? (or negative clinical trial to
provide evidences that the
The Methods outcomes of each intervention
The method of a research article usually are equal) and inferiority trial
(positive clinical trial, i.e. to proof
provides information about the study population, that the outcomes of
sample size, inclusion and exclusion criteria, as intervention
well as randomization method (since our are different
manuscript discusses about clinical trial, we will subjects; study design; sample size; analysis
focus on randomization method), blinding plan. Table 3 shows a list of question that
information and the analysis plan used in the relevant to the methods section.4,6,7
study. In brief, this part can generally divided Table 3. List of questions for the methods of clinical
journal article4
into four sub-components, i.e. population and

340
Vol 44 • Number 4 • October 2012 Critical appraisal on journal of clinical trials

Components Explanation while concealment is about how to conceal the

randomization table.4,6
Population and - Who are the target and
subjects accessible population in the Analysis plan usually will provide information
study?(target population refers to about what kind of statistical tests that will be used
the population that has become to analyse data and how the level of significance is
the objective of generalizing the
study, which has clinical and
used. Usually, the statistical analysis being used is
demographical components; while highly associated with hypothesis of the study; for
accessible population is part of example, Chi-square test is used when the study
target population wh ich has aims for comparing the proportion of unpaired
clinical, demographical, temporal
and spatial components) group. Moreover, we should also find out about
what kind of analysis method that has been used in
- What are the inclusion and
exclusion criteria? the study, either it is an analysis per protocol or
intention to treat (ITT) analysis. When the
Study design - What are the methods used for
randomization? (for example,
analysed subjects were those who had participated
simple, block, or stratified and completed the study, then the analysis per
randimisation etc). protocol would be appropriate. In contrast, ITT
- What tools that has been used for analysis is used for those who had been
randomization? (table or randomized although they had not complete the
computer)? study.4,6,8 Another analysis is the interim analysis,
- Who perform the randomization? Is i.e. any assessment of data performed before the
there any concealment? What kind number of all study subjects has been completed.
of concealment that has been
The analysis is usually used when there is a
performed?
suspicion that one of treatment is more superior
- Is there any blinding and how the than others and if the analysis has proven the
blinding was performed?
suspicion, the study should be stopped.4
Sample size - What is the formula of sample size
that (should be) used? (Is there any The Results
comparison between unpaired and The Result section of a clinical trial research
paired proportion, or between
unpaired and paired mean value?). article usually consists of the flow of the study and
the study data including primary and secondary
- What are the rationales for
determining sample size? (for outcomes. Table 4 shows a list of
example, does the study determine overall questions relevant to the results section.4,7
the degree of type I and II error and
the expected power?)
The flow of the study could be occassionally
found in the methods section. It is common to
Analysis plan - What kind of statistical tests that
have been used? Are they
present the flow of the study in diagram or flow
consistent with the hypothesis chart to provide easy reading about the process
and the aims of the study? involved in the study. Through the flow, we could
- What are the methods used for obtain some information about how many subjects
analysis? (analysis per protocol or have been recruited, how many subjects excluded
intention to treat?) from the study and how many eligible subjects.
- What computer program used for Other information that could be found is about
statistical analysis? how large is the percentage of eligible subjects and
- Is there any interim analysis that what is the percentage of subjects that were lost-to
has been planned for the study? follow up.
We should recognize some basic terms that Baseline data is a series of initial data prior to
are strongly associated with the methods the intervention. Our task here is to evaluate
section, which are randomization, blinding and whether the groups are comparable in regards of
concealment. Randomization refers to random demographical, socio-economic, and history of
allocation, i.e. a process of randomization to illness. If the baseline data are not comparable,
determine to which group a subject will be there are potentially confounding variables.
allocated. Randomization is useful to eliminate Therefore, researchers usually use stratification or
counfounding factors. Blinding is a procedure multivariate methods to control confounding
to mask information about the treatment given; variables in the analysis.
Table 4. List of questions for the results of clinical journal
article4

341
Murdani Abdullah Acta Med Indones-Indones J Intern
Med

Components Explanation demonstrated as p value <0.05, should always be

considered with its clinical significance.
Flow and Baseline - How many subjects are
data randomized for each group?
In clinical trials, cohort and case-control
studies, there are usually two comparable
- How many and what is the groups and the results are presented as a relative
percentage of subjects that
completed the study?
risk, i.e. a comparison of outcomes in the
intervention and control groups. If the outcomes
- How many and what is the
percentage of subjects that did
are measured as comparison of the occuring
not complete the study? event (for example the number of recovered
subjects compared to those who does not in a
- What are the reasons for
subjects being excluded from group), then such relative risk is known as odd
the study? ratio (OR). If the outcomes are calculated as a
- Are the baseline data
frequency, i.e. by comparing the study subjects
comparable? who experienced the event with the total
number of subjects in a group, then the relative
- What had been done by the
researcher when the baseline risk is defined as risk ratio (RR). When there is
data are not comparable? no difference found between groups, the value
Primary outcomes - What are the primary
of OR and RR is 1. Relative risk (either OR or
outcomes? RR) greater than 1 means that the number of
events are greater in the intervention group
- Are they statistically or clinicaly
significant? compared to the control group. In contrast,
when the relative risk is lesser than 1, the events
Secondary - What are the secondary
outcomes outcomes? (for example, side
in the intervention group is lesser.
effect) Results of the study would be more useful
Primary outcomes of the study provide when being presented as the risk differences. In
answers to primary research question; while this setting, we devide the proportion of events
secondary outcomes answer the additional in the control group with those in the
research question. Principally, when reading the intervention group. Risk difference is also
results of each study, we should perform clinical usually demonstrated as the number needed to
and statistical interpretation. The statistical treat (NNT) expressing how many subjects that
interpretation could have been done by evaluating should be treated during a period of time to
the p value and confidence interval of each achieve an outcome (treatment) or to prevent an
hypothesis test that has been performed (usually if event (prophylaxis/protection from disease).
p <0.05, the relative risk does not lied in the range Epidemiologically, NNT characterizes of how
of 1; proportional or mean value does not located many patients that would need health care in the
in the range of 0). Moreover, clinical interpretation form of treatment to reduce the number of
refers to comparison between the study outcome cases, which is presented as a final endpoint.
Table 5. How to calculate odd-ratio, risk ratio, and number needed to treat

Number
Number Risk of Number
of events Odds
of Odd ratio cure Risk ratio Risk difference needed to
(Ex. of cure
patients (frequency) treat
recovery)
Intervention 1,000 150 150/850 150/850 150/1000 150/1000 150 _ 100 = 0.05 1 / 0.05 =
= 1.59 = 1.5 1000 1000
Control 1,000 100 100/900 100/900 100/1000 100/1000 (=0.05%) 20

and effect size. Outcomes may appear as the value The following Table 5 illustrates those
of relative risk and proportional difference (if the parameters.
outcomes are categorical scale); mean difference
The Discussion
(when the outcomes are numerical scale); it may
also appear as hazard-ratio and incidence In the discussion section, we will find
difference (for outcomes of survival).4 Please information about the researcher’s steatement
notice that statistical significance, which is usually regarding validity, clinical significance and
applicability that related to the study results.

342
Vol 44 • Number 4 • October 2012 Critical appraisal on journal of clinical trials

However, the researchers are usually only 7. Juni P, Altman DG, Egger M. Systematic reviews in
discuss validity limited to three ascpects, which health care: Assessing the quality of controlled clinical
trials. BMJ. 2001;323:42–6.
are the power and limitation of study;
8. Burls A. What is critical appraisal? What is series?
comparing the study with other studies; and its London: Hayward Medical Communications; 2009..
explanation (biological plausabiity). A list of
questions relevant to the discussion section is
presented in Table 6 as follows.

Table 6. List of questions for the discussions of clinical

journal article4
Components
What are the power and limitation of study according to
the researcher’s point of view?
How is the comparison between the study results and
other studies?
How is the biological plausibility of the study results?
Can the results of study be generalized?
How is the clinical significance of the study results?
How is the applicability of study results according to the
researcher?

CONCLUSION
Critical appraisal is extremely needed as an
assessment of information found in scientific
journal articles before we are convinced to
apply the information as guidelines in clinical
practice. The level of reliability of a study
tremendously depends on the study design,
which places clinical trial as the highest in
rank. Critical appraisal evaluates all
components in a research article including the
introduction, method, results and discussion. A
good knowledge about how to evaluate each
component is very necessary to perform a good
critical appraisal. However, it should be
considered that statistical significance should
be compared with clinical significance.
REFERENCES
1. Oxford Centre for Evidence-Based Medicine. Levels of
evidence. 2012. [cited on June 10, 2012]. Available
online on http://www. cebm.net.
2. Belsey J. What is evidence-based medicine? London:
Hayward Medical Communications; 2009.
3. Pwee KH. What is this thing called EBM? Singapore Med
J. 2004;45:413-7.
4. Dahlan MS. Membaca dan menelaah jurnal uji klinis.
Jakarta: Salemba Medika; 2010.
5. Makela M, Witt K. How to read a paper: critical appraisal
of studies for application in healthcare. Singapore Med J.
2005;46(3):108-15.
6. Moore A, McQuay H. Clinical trials. Bandolier’s little
book of making sense of the medical evidence. Oxford:
Oxford University Press, 2006.

343