C H A P T E R 1

An Overview of Clinical Research:

The Lay of the Land

C H A P T E R C O N T E N T S

A Taxonomy of Clinical Research 2 Nonrandomised Trial: Penultimate Design? 7

What Studies Can and Cannot Do 4 Randomised Controlled Trial: Gold Standard 7
Descriptive Study 4 Measurement of Outcomes 8
Cross-Sectional Study: A Snapshot in Time 4 Confusing Fractions 8
Cohort Study: Looking Forward in Time 5 Measures of Association: Risky Business 9
Case-Control Study: Thinking Backwards 5 Conclusion 10
Variations on These Themes 6

Many clinicians report that they cannot read the medical literature critically. To address this dif-
ficulty, we provide a primer of clinical research for clinicians and researchers alike. Clinical research
falls into two general categories: experimental and observational, based on whether the investigator
assigns the exposures or not. Experimental trials can also be subdivided into two: randomised and
nonrandomised. Observational studies can be either analytical or descriptive. In this book, we dis-
tinguish between the two by this criterion: analytical studies feature a comparison (control) group,
whereas descriptive studies do not. Within analytical studies, cohort studies track people forward in
time from exposure to outcome. By contrast, case-control studies work in reverse, tracing back from
outcome to exposure. Cross-sectional studies are like a snapshot, which measures both exposure and
outcome at one time point. Descriptive studies cannot examine associations, a fact often forgotten or
ignored. Measures of association, such as relative risk or odds ratio, are the preferred way of expres-
sing results of dichotomous outcomes (e.g., sick versus healthy). Confidence intervals around these
measures indicate the precision of these results. Measures of association with confidence intervals
reveal the strength, direction, and plausible range of an effect, as well as the likelihood of chance
occurrence. By contrast, p values address only chance. Testing null hypotheses at a p value of
0.05 has no basis in medicine and should be discouraged.

Clinicians today are in a bind. Increasing demands on their time are squeezing out opportunities to
stay abreast of the literature, much less read it critically. PubMed currently catalogues 30,000 jour-
nals, and its citations include more than 27 million entries. Results of several studies indicate an
inverse relation between quality of care and time since graduation from medical school;1 older cli-
nicians consistently have a harder time keeping up.2 In many jurisdictions, attendance at a specified
number of hours of continuing medical education (CME) courses is mandatory to maintain a
licence to practice. Continuing medical education courses have at best a modest effect on the quality
of care, and such courses alone appear unlikely to affect complex clinical behaviours.3 This defi-
ciency of traditional CME offerings emphasises the importance of self-directed learning through

1
2 INTRODUCTION

reading. However, many clinicians lack the requisite skills to read the medical literature critically.4,5
Scientific illiteracy6 and innumeracy5 remain major failings of medical education.
This book on research methods is designed for busy clinicians and active researchers. The needs
of clinicians predominate; hopefully, this primer will produce more critical and thoughtful con-
sumers of research, and thus better practitioners. The needs of clinicians overlap with those of
researchers throughout the chapters, but that overlap becomes most pronounced in the discussion
of randomised controlled trials. For readers to assess randomised trials accurately, they should
understand the relevant guidelines on the conduct of trials; these guidelines have been derived from
empirical methodological research.
The disproportionate coverage of randomised controlled trials is intentional; randomised con-
trolled trials are the gold standard in clinical research. Randomised controlled trials help to elim-
inate bias, and research has identified the important methodological elements of trials that minimise
bias.7,8 Finally, because trials are deemed more credible than observational studies,9,10 clinicians
might be more likely to act on their results than on those of other study types; hence, investigators
need to ensure that trials are done well and reported well. To start, we provide a brief overview of
research designs and discuss some of the common measures used.

A Taxonomy of Clinical Research

Analogous to biological taxonomy, a simple hierarchy can be used to categorise most studies
(Panel 1.1).11 To do so, however, the study design must be known. As in biology, anatomy dictates
physiology. The anatomy of a study determines what it can and cannot do. A difficulty that readers
encounter is that authors sometimes do not report the study type or provide sufficient detail to figure
it out. A related problem is that authors sometimes incorrectly label the type of research done.
Examples include calling nonrandomised controlled trials randomised,12 and labelling nonconcur-
rent cohort studies case-control studies.13-15 The adjective ‘case-controlled’ is also sometimes (inap-
propriately) applied to any study with a comparison group. The media compound the confusion by
implying causation16 (when researchers cautiously reported only an association).17
Biology has animal and plant kingdoms. Similarly, clinical research has two large kingdoms: exper-
imental and observational research. Fig. 1.1 shows that one can quickly decide the research kingdom
by noting whether the investigators assigned the exposures (e.g., treatments) or whether they observed

PANEL 1.1 ■ Rating Clinical Evidence: Assessment System of the US Preventive

Services Task Force11

Quality of Evidence
I Evidence from at least one properly designed randomised controlled trial.
II-1 Evidence obtained from well-designed controlled trials without randomisation.
II-2 Evidence from well-designed cohort or case-control studies, preferably from more than one
centre or research group.
II-3 Evidence from multiple time series with or without the intervention. Important results in
uncontrolled experiments (such as the introduction of penicillin treatment in the 1940s) could
also be considered as this type of evidence.
III Opinions of respected authorities, based on clinical experience, descriptive studies, or reports of
expert committees.
Strength of Recommendations
A Strongly recommends
B Recommends
C Makes no recommendation for or against
D Recommends against
I Concludes evidence insufficient to recommend for or against
1—AN OVERVIEW OF CLINICAL RESEARCH: THE LAY OF THE LAND 3

Did investigator
assign exposures?

Yes No

Experimental study Observational study

Random allocation? Comparison group?

Yes No Yes No

Analytical Descriptive
Non-
Randomised study study
randomised
controlled controlled
trial trial Direction?

Exposure Outcome Exposure and

outcome at
the same time
Exposure Outcome

Case- Cross-
Cohort
control sectional
study
study study

Fig. 1.1 Algorithm for classification of types of clinical research.

usual clinical practice.18-20 At the top of the hierarchy, for experimental studies one needs to distin-
guish whether the exposures were assigned by a truly random technique (with concealment of the
upcoming assignment from those involved) or whether some other allocation scheme was used, such
as alternate assignment.21 Many reports fail to include this critical information.22
With observational studies, which dominate the literature, especially in lower-impact journals,23
the next step is to ascertain whether the study has a comparison or control group. If so, the study is
termed analytical.24 If not, it is a descriptive study (see Fig. 1.1). If the study is analytical, the tem-
poral direction needs to be identified. If the study determines both exposures and outcomes at one
time point, it is termed cross-sectional. An example would be measurement of serum cholesterol of
patients admitted to a hospital with myocardial infarction versus that of their next-door neighbour.
This type of study provides a snapshot of the population of sick and well at one time point.25 A
weakness of cross-sectional studies is that the temporal sequence may be unclear: did the exposure
precede the outcome? In addition, cross-sectional studies are inappropriate for rare diseases or those
that resolve quickly; severe diseases causing early death can mean that those studied are not repre-
sentative of all those with a given disease.19
If the study begins with an exposure (e.g., oral contraceptive use) and follows women for years to
measure outcomes (e.g., ovarian cancer), then it is deemed a cohort study. Cohort studies can be
either concurrent or nonconcurrent. By contrast, if the analytical study begins with an outcome
(e.g., ovarian cancer) and looks back in time for an exposure, such as use of oral contraceptives, then
the study is a case-control study.
Studies without comparison groups are called descriptive studies.24 At the bottom of the
research hierarchy is the case report. Indeed, the literature is replete with reported oddities.26
4 INTRODUCTION

When more than one patient is described, it becomes a case-series report. Some new diseases first
appear in the literature this way.27

What Studies Can and Cannot Do

DESCRIPTIVE STUDY
Starting at the bottom of the research hierarchy, descriptive studies are often the first foray into a new
area of medicine, a first ‘toe in the water’. Investigators do descriptive studies to describe the frequency,
natural history, and possible determinants of a condition.18,19 The results of these studies show how
many people develop a disease or condition over time, describe the characteristics of the disease and
those affected, and generate hypotheses about the cause of the disease. These hypotheses can be
assessed through more rigorous research, such as analytical studies or randomised controlled trials.
An example of a descriptive study would be the early reports of legionnaires’ disease28 and toxic-shock
syndrome.29 An important caveat (often forgotten or intentionally ignored) is that descriptive studies,
which do not have a comparison group, do not allow assessment of associations.30 Only comparative
studies (both analytical and experimental) enable assessment of possible causal associations.

CROSS-SECTIONAL STUDY: A SNAPSHOT IN TIME

Sometimes termed a frequency survey or a prevalence study,24 cross-sectional studies are done to
examine the presence or absence of disease and the presence or absence of an exposure at a particular
time. Thus prevalence, not incidence, is the focus. As both outcome and exposure are ascertained at
the same time (Fig. 1.2), the temporal relation between the two can be unclear. For example, assume
that a cross-sectional study finds obesity to be more common among women with arthritis than
among those without this condition. Did the extra weight load on joints lead to arthritis, or did
women with arthritis become involuntarily inactive and then obese? This ‘chicken-egg’ question
is unanswerable in a cross-sectional study.

Cohort study
Exposure Outcome

Case-control study
Exposure Outcome

Cross-sectional study
Exposure

Outcome

Time
Fig. 1.2 Schematic diagram showing temporal direction of three study designs.
1—AN OVERVIEW OF CLINICAL RESEARCH: THE LAY OF THE LAND 5

COHORT STUDY: LOOKING FORWARD IN TIME

Cohort studies proceed in a logical sequence from exposure to outcome (see Fig. 1.2). Hence, this
type of research is easier to understand than case-control studies. Investigators identify a group with
an exposure of interest and another group or groups without the exposure. The investigators then
follow the exposed and unexposed groups forward in time to determine outcomes. If the exposed
group develops a higher incidence of the outcome than the unexposed, then the exposure is asso-
ciated with an increased risk of the outcome, and vice versa.
Cohort studies can be prospective, retrospective, or both. In a prospective cohort study, the
exposed and unexposed are identified and followed forward in time to outcomes. In a retrospective
cohort study, the investigator goes back in time through medical records to identify exposure
groups, then tracks them to outcomes through existing medical records. In some disciplines, ret-
rospective cohort studies are more common than prospective cohort studies. As described in
Chapter 4, ambidirectional cohort studies look backward and forward in time.
The cohort study has important strengths and weaknesses. Because exposure is identified at the
outset, one can usually determine that the exposure preceded the outcome. Recall bias is less of a
concern than in the case-control study. The cohort study enables calculation of true incidence rates,
relative risks, and attributable risks. However, for the study of rare events or events that take years to
develop, this type of research design can be slow to yield results and thus prohibitively expensive.
Nonetheless, several famous, large cohort studies31,32 continue to provide important information.

CASE-CONTROL STUDY: THINKING BACKWARDS

Case-control studies work backwards. Because thinking in this direction is not intuitive for clini-
cians, case-control studies are widely misunderstood. Indeed, in one sample from the rehabilitation
literature, 97% of research articles designated ‘case-control’ were mislabelled.33
Starting with an outcome, such as disease, this type of study looks backward in time for expo-
sures that might be related to the outcome. As shown in Fig. 1.2, investigators define a group with
an outcome (e.g., ovarian cancer) and a group without the outcome (controls). Then, through chart
reviews, interviews, electronic medical records, or other means, the investigators ascertain the prev-
alence (or amount) of exposure to a risk factor (e.g., oral contraceptives, ovulation-induction drugs)
in both groups. If the prevalence of the exposure is higher among cases than among controls, then
the exposure is associated with an increased risk of the outcome.
Case-control studies are especially useful for outcomes that are rare or that take a long time to
develop, prototypes being cardiovascular disease and cancer. These studies often require less time,
effort, and money than would cohort studies. The Achilles heel of case-control studies is choosing
an appropriate control group, discussed further in Chapter 6. Controls should be similar to cases in
all important respects except for not having the outcome in question. Inappropriate control groups
have ruined many case-control studies and caused much harm. Additionally, recall bias (better rec-
ollection of exposures among the worried cases than among the healthy controls) is inevitable in
studies that rely on memory. Because the case-control study samples on cases and controls, not
on exposures, investigators cannot calculate incidence rates, relative risks, or attributable risks.
Instead, odds ratios are the measure of association used; when the outcome is uncommon (e.g., most
cancers) the odds ratio provides a good proxy for the true relative risk.34
Investigations of outbreaks of food-borne diseases often use case-control studies. On a cruise
ship, the entire universe of those at risk is known. Those with vomiting and diarrhoea (cases)
are asked about food exposures, as are a sample of those not ill (controls). If a higher proportion
of those who are ill reports having eaten a specific food than those who are well, the food becomes
suspect. In 50 outbreaks associated with cruise ships from 1970 to 2003, the most common culprit
was found to be seafood (28% of outbreaks).35
6 INTRODUCTION

VARIATIONS ON THESE THEMES

Nested Case-Control Studies
Here, a group of persons being studied is followed forward in time until outcomes occur. Instead of
analysing everyone in the group in usual cohort fashion, those with the outcome are deemed cases,
and a sample of those without the disease become controls.36 Then, the analysis proceeds in typical
case-control fashion. The smaller case-control study is ‘nested’ within a larger cohort under
scrutiny.24
Why analyse a cohort study this way? This approach may be useful when determining the expo-
sure involves an expensive, painful, invasive test or ponderous interview. Determining exposure
information for all participants might be impractical or prohibitively expensive. All cases have expo-
sure determined, but only a small sample of all healthy persons (controls) need this done. A study of
inflammatory markers and postpartum depression is illustrative. A prospective cohort of women
being followed through pregnancy had blood samples collected and banked at specified times.
Sixty-three women with depression had a sophisticated protein panel run on their blood sample
to examine 92 inflammation markers. The same panel was run on blood from 228 controls without
depression.37 Performing this extensive testing on all women in the cohort would have been costly.

Case-Cohort Studies
In this modification of a case-control study, ‘…controls are drawn from the same cohort as the cases
regardless of their disease status. Cases of the disease of interest are identified, and a sample of the
entire starting cohort (regardless of their outcomes) forms the controls’24 Here, one control group
can be used for multiple groups of cases, increasing the efficiency of the exploration. Controls are
randomly selected from the population at risk; these controls are called the ‘reference subcohort’.38
In a traditional case-control study, researchers would need to recruit a different control group for
each group of cases being studied; here, one control group can be recycled repeatedly.
A case-cohort study examined risk factors for hospitalisation after dog-bite injury.39 The cohort
consisted of 1384 dog-bite patients (Fig. 1.3) seen at one hospital emergency department; 111 of
these who were admitted formed the case group. A simple random sample of the other patients

Fig. 1.3 Dog bite.

1—AN OVERVIEW OF CLINICAL RESEARCH: THE LAY OF THE LAND 7

comprised the control subcohort (221 patients). As noted previously, controls were selected inde-
pendent of their disease status; by chance, 21 patients randomly chosen as controls were themselves
cases (admitted to the hospital). Infected bites and bites to multiple sites were most strongly related
to the risk of being hospitalised.

Case-Crossover Studies
This study type can be used to assess adverse drug reactions or events rapidly triggered by an expo-
sure. For example, a case with an adverse event is queried about exposures in the prior hour, deemed
the hazard window.25 The control interval might be the same time window in the same patient
1 day earlier. This helps to control for confounding by indication, because each study participant
serves as both the case and the control.40 An example would be studying the association between
sedatives and car wrecks.41
The possibility that chiropractic manipulation might precipitate carotid artery stroke prompted
a case-crossover study in Canada.42 All incident strokes over a 9-year period were the cases; each
served as his or her own control. The exposure windows used were 1, 3, 7, and 14 days before the
stroke. No significant differences in risk between visits to chiropractors and primary care providers
were evident, suggesting no role of spinal manipulation in triggering stroke.

Case-Time-Control Studies
This approach is a variant of the case-crossover design in which a traditional control group provides
the background exposure history. This type of study is defined as ‘a study in which exposure of cases
and controls during one period is compared in matched-pair analyses to their own exposure during
another period of similar length’.24 This approach is used in pharmacological studies to study brief
exposures with acute effects, such as nonsteroidal antiinflammatory drugs and cardiac arrest outside
of a hospital.43 This approach helps to control for time trends in exposures.40

NONRANDOMISED TRIAL: PENULTIMATE DESIGN?

Some experimental trials do not randomly allocate participants to exposures (e.g., treatments or
prevention strategies). Instead of using truly random techniques, investigators may use methods
that fall short of the mark (e.g., alternate assignment).44 The US Preventive Services Task Force11
designates this research design as class II-1, indicating less scientific rigour than randomised trials,
but more than analytical studies (see Panel 1.1).
After the investigators have assigned participants to treatment groups, the way a nonrandomised
trial is done and analysed resembles that of a cohort study. The exposed and unexposed are followed
forward in time to ascertain the frequency of outcomes. Advantages of a nonrandomised trial
include use of a concurrent control group and uniform ascertainment of outcomes for both groups.
However, selection bias can occur, because allocation concealment may be impossible to maintain.

RANDOMISED CONTROLLED TRIAL: GOLD STANDARD

The randomised controlled trial is the only known way to avoid selection and confounding biases in
clinical research. This design approximates the controlled experiment of basic science. It resembles
the cohort study in several respects, with the important exception of randomisation of participants
to exposures (see Fig. 1.2).
The hallmark of randomised controlled trials is assignment of participants to exposures (e.g.,
treatments) purely by the play of chance. When properly implemented, random allocation precludes
selection bias. Randomised controlled trials reduce the likelihood of bias in determination of out-
comes. Trials feature uniform diagnostic criteria for outcomes and, often, blinding of treatment
assignment, thus reducing the potential for information bias. A unique strength of this study design
8 INTRODUCTION

is that it eliminates confounding bias, both known and unknown (discussed in Chapter 3). Trials
tend to be statistically efficient. If properly designed and done, a randomised controlled trial is likely
to be free of bias and is thus especially useful for examination of small or moderate effects. In obser-
vational studies, bias can easily account for weak associations (i.e., small differences).45
Randomised controlled trials have drawbacks, too. External validity is a concern. Whereas the
randomised controlled trial, if properly done, has internal validity (i.e., it measures what it sets out to
measure), it might not have external validity. This term indicates the extent to which results can be
generalised to the broader community.20 Unlike the observational study, the randomised controlled
trial includes only volunteers who pass through a screening process before inclusion. Those who
volunteer for trials tend to be different from the population of those with the condition; only a tiny
fraction of those with a condition find their way into trials.46 Another limitation is that a rando-
mised controlled trial cannot be used in some instances, because intentional exposure to harmful
substances (e.g., toxins, bacteria, or other noxious substances) would be unethical. As with cohort
studies, the randomised controlled trial can be prohibitively expensive. Indeed, the cost of large
trials can run into the hundreds of millions of US dollars.47

Measurement of Outcomes
CONFUSING FRACTIONS
Identification and quantification of outcomes is the business of research. However, slippery termi-
nology often complicates matters for investigators and readers alike. For example, the term ‘rate’ (as
in ‘maternal mortality rate’) has been misused in textbooks and journal articles for decades. Addi-
tionally, ‘rate’ is often used interchangeably with proportions and ratios.24 Fig. 1.4 presents a simple
approach to classification of these common terms.
A ratio is a value obtained by dividing one number by another.24 These two numbers can be
either related or unrelated. This feature (i.e., relatedness of numerator and denominator) separates
ratios into two groups: those in which the numerator is included in the denominator (e.g., rate and
proportion) and those in which it is not.
A rate measures the frequency of an event in a population, usually over a period of time.
As shown in Fig. 1.4, the numerator (those with the outcome) of a rate must be contained in

Ratio

Is numerator included
in denominator?

Yes No

Is time included
in denominator?

Yes No

Measure: Rate Proportion Ratio

Example: Incidence rate Prevalence rate Maternal mortality ratio

Fig. 1.4 Algorithm for distinguishing rates, proportions, and ratios.

1—AN OVERVIEW OF CLINICAL RESEARCH: THE LAY OF THE LAND 9

the denominator (those at risk of the outcome). Although all ratios feature a numerator and denom-
inator, rates have two distinguishing characteristics: time and a multiplier. Rates indicate the time
during which the outcomes occur and a multiplier, commonly to a base ten, to yield whole numbers.
For example, the age-adjusted incidence of all unprovoked seizures in Minnesota is 61 per 100,000
person-years. In contrast, the prevalence of epilepsy there is 6.8 per 1000 population.48 The latter
measure is the total number of persons with a condition at a point in time divided by the population
at risk at that point in time (or midway in the time interval).24
Proportion is often used synonymously with rate, but the former does not have a time compo-
nent. Like a rate, a proportion is a ratio with the numerator contained in the denominator.24
Because the numerator and denominator have the same units, these divide out, leaving a dimen-
sionless quantity (a number without units). An example of a proportion is prevalence (e.g., 27 of 100
at risk have hay fever). This number indicates how many of a population at risk have a condition at a
particular time (here, 27%); as documentation of new cases over time is not involved, prevalence is
more properly considered a proportion than a rate.
Although all rates and proportions are ratios, the opposite is not true. In some ratios, the numer-
ator is not included in the denominator. The most notorious example may be the venerable maternal
mortality ratio. The definition includes women who die of pregnancy-related causes in the numer-
ator, and women with live births (usually 100,000) in the denominator. However, not all those in
the numerator are included in the denominator (e.g., a woman who dies of an ectopic pregnancy at
7 weeks’ gestation cannot be in the denominator). Thus this ‘rate’ is actually a ratio, not a rate, a fact
still not widely understood.49

MEASURES OF ASSOCIATION: RISKY BUSINESS

Relative risk (also termed the risk ratio) is another useful ratio: the frequency of outcome in the
exposed divided by the frequency of outcome in the unexposed. If the frequency of the outcome
is the same in both groups, then the ratio is 1.0, indicating no association between exposure
and outcome. By contrast, if the outcome is more frequent in those exposed, then the ratio will
be greater than 1.0, implying an increased risk associated with exposure. Conversely, if the fre-
quency of disease is less among the exposed, then the relative risk will be less than 1.0, implying
a protective effect.
Also known as the cross-products ratio or relative odds,24 the odds ratio has different meanings
in different settings. In case-control studies, this ratio is the usual measure of association. It indi-
cates the odds of exposure among the case group divided by the odds of exposure among controls.
If cases and controls have equal odds of having the exposure, the odds ratio is 1.0, indicating no
association. If the cases have a higher odds of exposure than the controls, then the ratio is greater
than 1.0, implying an increased risk associated with exposure. Conversely, odds ratios less than 1.0
indicate a protective effect.
An odds ratio can also be calculated for cross-sectional, cohort, and randomised controlled stud-
ies. Here, the disease-odds ratio is the ratio of the odds in favour of disease in the exposed versus that
in the unexposed. In this context, the odds ratio has some appealing statistical features when studies
are aggregated in meta-analysis, but the odds ratio does not indicate the relative risk when the pro-
portion with the outcome is greater than 5% to 10%.34 Stated alternatively, the odds ratio is always a
valid measure of association, but it is not a good proxy for the relative risk unless the outcome is
uncommon (‘rare disease assumption’).
The confidence interval (CI) reflects the precision of study results. The interval provides a range
of plausible values for a point estimate, such as a proportion, relative risk, or odds ratio. The con-
fidence interval has a specified probability of containing the true value for the entire population from
which the study sample was taken. Although 95% CIs are the most commonly used, others, such as
90%, are sometimes seen.50 The wider the confidence interval, the less precision exists in the result,
10 INTRODUCTION

and vice versa. For relative risks and odds ratios, when the 95% CI does not include 1.0, the dif-
ference is significant at the usual 0.05 level. However, use of this feature of confidence intervals as a
back-door means of hypothesis testing is inappropriate.51

Conclusion
Understanding what kind of study has been done is a prerequisite to thoughtful reading of research.
Clinical research can be divided into experimental and observational; observational studies are fur-
ther categorised into those with and without a comparison group. Only studies with comparison
groups allow investigators to assess possible causal associations, a fact often forgotten or ignored.
Dichotomous outcomes of studies should be reported as measures of association with confidence
intervals; testing null hypotheses at arbitrary p values of 0.05 has no basis in medicine.

References
1. Goulet, F., Hudon, E., Gagnon, R., Gauvin, E., Lemire, F., Arsenault, I., 2013. Effects of continuing
professional development on clinical performance: results of a study involving family practitioners in Que-
bec. Can. Fam. Physician 59, 518–525.
2. Decker, S.L., Jamoom, E.W., Sisk, J.E., 2012. Physicians in nonprimary care and small practices and those
age 55 and older lag in adopting electronic health record systems. Health Aff. (Millwood) 31, 1108–1114.
3. Forsetlund, L., Bjorndal, A., Rashidian, A., et al., 2009. Continuing education meetings and workshops:
effects on professional practice and health care outcomes. Cochrane Database Syst. Rev. CD003030.
4. Windish, D.M., Huot, S.J., Green, M.L., 2007. Medicine residents’ understanding of the biostatistics and
results in the medical literature. JAMA 298, 1010–1022.
5. Rao, G., Kanter, S.L., 2010. Physician numeracy as the basis for an evidence-based medicine curriculum.
Acad. Med. 85, 1794–1799.
6. Hadley, J.A., Wall, D., Khan, K.S., 2007. Learning needs analysis to guide teaching evidence-based med-
icine: knowledge and beliefs amongst trainees from various specialities. BMC Med. Educ. 7, 11.
7. Schulz, K.F., Altman, D.G., Moher, D., 2010. CONSORT 2010 statement: updated guidelines for report-
ing parallel group randomised trials. BMJ 340, c332.
8. Moher, D., Hopewell, S., Schulz, K.F., et al., 2010. CONSORT 2010 explanation and elaboration:
updated guidelines for reporting parallel group randomised trials. J. Clin. Epidemiol. 63, e1–37.
9. Martinez, M.E., Marshall, J.R., Giovannucci, E., 2008. Diet and cancer prevention: the roles of observation
and experimentation. Nat. Rev. Cancer 8, 694–703.
10. Albert, R.K., 2013. “Lies, damned lies …” and observational studies in comparative effectiveness research.
Am. J. Respir. Crit. Care Med. 187, 1173–1177.
11. Harris, R.P., Helfand, M., Woolf, S.H., et al., 2001. Current methods of the US Preventive Services Task
Force: a review of the process. Am. J. Prev. Med. 20, 21–35.
12. Wu, T., Li, Y., Bian, Z., Liu, G., Moher, D., 2009. Randomized trials published in some Chinese journals:
how many are randomized? Trials 10, 46.
13. Grimes, D.A., 2009. “Case-control” confusion: mislabeled reports in obstetrics and gynecology journals.
Obstet. Gynecol. 114, 1284–1286.
14. Hellems, M.A., Kramer, M.S., Hayden, G.F., 2006. Case-control confusion. Ambul. Pediatr. 6, 96–99.
15. Caro, J.J., Huybrechts, K.F., 2009. Case-control studies in pharmacoeconomic research: an overview.
Pharmacoeconomics 27 (8), 627–634.
16. Rossman S. Diet soda can increase risk of dementia and stroke, study finds. https://www.usatoday.com/story/
news/nation-now/2017/04/21/diet-soda-can-increase-risk-dementia-and-stroke-study-finds/100736786/,
accessed 23 April 2017.
17. Pase, M.P., Himali, J.J., Beiser, A.S., et al., 2017. Sugar- and artificially sweetened beverages and the risks
of incident stroke and dementia: a prospective cohort study. Stroke 48, 1139–1146.
18. Meyer, A.M., Wheeler, S.B., Weinberger, M., Chen, R.C., Carpenter, W.R., 2014. An overview of
methods for comparative effectiveness research. Semin. Radiat. Oncol. 24, 5–13.
19. DiPietro, N.A., 2010. Methods in epidemiology: observational study designs. Pharmacotherapy
30, 973–984.
1—AN OVERVIEW OF CLINICAL RESEARCH: THE LAY OF THE LAND 11

20. Stang, A., 2008. Appropriate epidemiologic methods as a prerequisite for valid study results. Eur. J. Epi-
demiol. 23, 761–765.
21. Hill, C.L., LaValley, M.P., Felson, D.T., 2002. Discrepancy between published report and actual conduct
of randomized clinical trials. J. Clin. Epidemiol. 55, 783–786.
22. Strech, D., Soltmann, B., Weikert, B., Bauer, M., Pfennig, A., 2011. Quality of reporting of randomized
controlled trials of pharmacologic treatment of bipolar disorders: a systematic review. J. Clin. Psychiatry
72, 1214–1221.
23. Kuroki, L.M., Allsworth, J.E., Peipert, J.F., 2009. Methodology and analytic techniques used in clinical
research: associations with journal impact factor. Obstet. Gynecol. 114, 877–884.
24. Porta, M., 2014. A Dictionary of Epidemiology. Oxford University Press, New York.
25. Hartung, D.M., Touchette, D., 2009. Overview of clinical research design. Am. J. Health Syst. Pharm.
66, 398–408.
26. Goldberg, H.R., Allen, L., Kives, S., 2017. Fetiform teratoma in the ovary of a 7-year-old girl: a case report.
J. Pediatr. Adolesc. Gynecol. 30, 256–258.
27. Pneumocystis pneumonia—Los Angeles, 1981. MMWR Morb. Mortal. Wkly Rep. 30, 250–252.
28. Keys, T.F., 1977. A sporadic case of pneumonia due to legionnaires disease. Mayo Clin. Proc. 52, 657–660.
29. McKenna, U.G., Meadows 3rd, J.A., Brewer, N.S., Wilson, W.R., Perrault, J., 1980. Toxic shock syn-
drome, a newly recognized disease entity. Report of 11 cases. Mayo Clin. Proc. 55, 663–672.
30. Caillouette, J.C., Koehler, A.L., 1987. Phasic contraceptive pills and functional ovarian cysts. Am. J.
Obstet. Gynecol. 156, 1538–1542.
31. Wei, E.K., Colditz, G.A., Giovannucci, E.L., et al., 2017. A comprehensive model of colorectal
cancer by risk factor status and subsite using data from the nurses’ health study. Am. J. Epidemiol.
185, 224–237.
32. Iversen, L., Sivasubramaniam, S., Lee, A.J., Fielding, S., Hannaford, P.C., 2017. Lifetime cancer risk and
combined oral contraceptives: the Royal College of General Practitioners’ Oral Contraception Study.
Am. J. Obstet. Gynecol. 216, 580.e1–580.e9.
33. Mayo, N.E., Goldberg, M.S., 2009. When is a case-control study not a case-control study? J. Rehabil. Med.
41, 209–216.
34. Grimes, D.A., Schulz, K.F., 2008. Making sense of odds and odds ratios. Obstet. Gynecol. 111,
423–426.
35. Rooney, R.M., Cramer, E.H., Mantha, S., et al., 2004. A review of outbreaks of foodborne disease asso-
ciated with passenger ships: evidence for risk management. Public Health Rep. 119, 427–434.
36. Etminan, M., 2004. Pharmacoepidemiology II: the nested case-control study—a novel approach in phar-
macoepidemiologic research. Pharmacotherapy 24, 1105–1109.
37. Brann, E., Papadopoulos, F., Fransson, E., et al., 2017. Inflammatory markers in late pregnancy in
association with postpartum depression—A nested case-control study. Psychoneuroendocrinology
79, 146–159.
38. Checkoway, H., Pearce, N., Kriebel, D., 2007. Selecting appropriate study designs to address specific
research questions in occupational epidemiology. Occup. Environ. Med. 64, 633–638.
39. Rhea, S., Weber, D.J., Poole, C., Cairns, C., 2014. Risk factors for hospitalization after dog bite injury: a
case-cohort study of emergency department visits. Acad. Emerg. Med. 21, 196–203.
40. Schneeweiss, S., Sturmer, T., Maclure, M., 1997. Case-crossover and case-time-control designs as alter-
natives in pharmacoepidemiologic research. Pharmacoepidemiol. Drug Saf. 6 (Suppl 3), S51–S59.
41. Lu, C.Y., 2009. Observational studies: a review of study designs, challenges and strategies to reduce
confounding. Int. J. Clin. Pract. 63, 691–697.
42. Cassidy, J.D., Boyle, E., Cote, P., Hogg-Johnson, S., Bondy, S.J., Haldeman, S., 2017. Risk of carotid
stroke after chiropractic care: a population-based case-crossover study. J. Stroke Cerebrovasc. Dis.
26, 842–850.
43. Sondergaard, K.B., Weeke, P., Wissenberg, M., et al., 2017. Non-steroidal anti-inflammatory drug use is
associated with increased risk of out-of-hospital cardiac arrest: a nationwide case-time-control study. Eur.
Heart J. Cardiovasc. Pharmacother. 3, 100–107.
44. Sharma, P.K., Yadav, T.P., Gautam, R.K., Taneja, N., Satyanarayana, L., 2000. Erythromycin in pityriasis
rosea: A double-blind, placebo-controlled clinical trial. J. Am. Acad. Dermatol. 42, 241–244.
45. Grimes, D.A., Schulz, K.F., 2012. False alarms and pseudo-epidemics: the limitations of observational epi-
demiology. Obstet. Gynecol. 120, 920–927.
12 INTRODUCTION

46. Rothwell, P.M., 2005. External validity of randomised controlled trials: “to whom do the results of this trial
apply?”. Lancet 365, 82–93.
47. Roth, J.A., Etzioni, R., Waters, T.M., et al., 2014. Economic return from the Women’s Health Initiative
estrogen plus progestin clinical trial: a modeling study. Ann. Intern. Med. 160, 594–602.
48. Logroscino, G., Hesdorffer, D.C., 2005. Methodologic issues in studies of mortality following epilepsy:
measures, types of studies, sources of cases, cohort effects, and competing risks. Epilepsia 46 (Suppl
11), 3–7.
49. MacDorman, M.F., Declercq, E., Cabral, H., Morton, C., 2016. Recent increases in the U.S. maternal
mortality rate: disentangling trends from measurement issues. Obstet. Gynecol. 128, 447–455.
50. Catton, C.N., Lukka, H., Gu, C.S., et al., 2017. Randomized trial of a hypofractionated radiation regimen
for the treatment of localized prostate cancer. J. Clin. Oncol. 35, 1884–1890.
51. Sterne, J.A., Smith, G.D., 2001. Sifting the evidence—what’s wrong with significance tests? BMJ
322, 226–231.