Types of study designs:

from descriptive studies to

randomized controlled trials

Kirsten Bibbins-Domingo, PhD, MD

Assistant Professor of Medicine and of
Epidemiology and Biostatistics
University of California, San Francisco
 Objectives
• To understand the difference between
• descriptive and analytic studies
• To identify the hierarchy of study
• designs, and the strengths and
• weakness of each design
• To be able to apply different study
• designs to the same research question
 Types of Studies
Descriptive Studies
• Observational Analytic Studies
• Cross Sectional studies
Case Control studies
Cohort studies

Experimental Studies
• Randomized controlled trials
Hierarchy of Study Types
 Descriptive studies
Getting a “lay of the land”
Surveys (NHIS, MCBS)
• “How many men in the U.S. filled Viagra prescriptions in
2004?”
Describing a novel phenomena
• Case reports or case series
• Viagra-associated serous macular
• detachment.
• Sildenafil-associated nonarteritic anterior ischemic optic
neuropathy
 Descriptive studies
Cannot establish causal relationships
• Still play an important role in describing trends and
generating hypotheses about novel associations
The start of HIV/AIDS research
• Squamous cell carcinoma in sexual partner of Kaposi sarcoma
patient. Lancet. 1982 Jan 30;1(8266):286.
• New outbreak of oral tumors, malignancies and infectious
diseases strikes young male homosexuals. CDA J. 1982
Mar;10(3):39-42.
• AIDS in the "gay" areas of San Francisco. Lancet. 1983 Apr
23;1(8330):923-4
 Analytic Studies
Attempt to establish a causal link between a
predictor/risk factor and an outcome.
You are doing an analytic study if you have any of
the following words in your research question:
• greater than, less than, causes, leads to,
compared with, more likely than, associated
with, related to, similar to, correlated with
 Research Question
Is the regular consumption of Red Bull
associated with improved academic
performance among U.S. medical students?
 Rationale
“functional drink” designed for periods of
mental and physical exertion.
• performance, concentration, memory,
• reaction time, vigilance, and emotional
balance
Taurine + glucuronolactone + caffeine
 Background
• Alford C, Cox H, Wescott R. The effects of red bull energy drink on human
performance and mood. Amino Acids. 2001;21(2):139-50.
• Warburton DM, Bersellini E, Sweeney E. An evaluation of a caffeinated taurine
drink on mood, memory and information processing in healthy volunteers
without caffeine abstinence. Psychopharmacology (Berl). 2001 Nov;158(3):322-8.
• Seidl R, Peyrl A, Nicham R, Hauser E. A taurine and caffeine-containing drink
stimulates cognitive performance and well-being. Amino Acids. 2000;19(3-4):635-
42.
• Horne JA, Reyner LA. Beneficial effects of an "energy drink" given to sleepy
drivers. Amino Acids. 2001;20(1):83-9.
• Kennedy DO, Scholey AB. A glucose-caffeine 'energy drink'
ameliorates subjective and performancedeficits during prolonged
cognitive demand. Appetite. 2004 Jun;42(3):331-3.
 Great idea, but how do you get
started….

Interesting, novel, and relevant, but…

• You only have 25,000 dollars to start
investigating this question.
• What is feasible?
 Study Design #1
Cross-sectional study of UCSF medical students
taking USMLE Step 2
Questionnaire administered when registering for
USMLE 2
• Primary predictor: self-report of >3 cans Red Bull
per week for the previous year
• Covariates: Age, sex, undergraduate university,
place of birth
Outcome: Score on USMLE Step 2
 Cross-sectional Study:
Descriptive value:
• How many UCSF medical students drink Red Bull?
• What is the age and sex distribution of UCSF medical students who drink Red
Bull?
Analytic value:
• Is there an association between regular Red Bull consumption and test scores
among UCSF med students?
• Univariate
• Multivariate (controlling for “confounders”)
Other cross-sectional surveys:
• AAMC
• California Health Interview Survey (NHIS, CHIS)
• National Health and Nutrition Exam Survey (NHANES)
 Cross-sectional Study: Pluses
• Prevalence (not incidence)
+ Fast/Inexpensive - no waiting!
+ No loss to follow up
+ Associations can be studied
Measures of association
 Cross-sectional study: minuses
Cannot determine causality
 Cross-sectional study: minuses
Cannot determine causality
• ACE inhibitor use and hospitalization rates
among those with heart failure
• Heart failure patients with a documented DNR
status and mortality
 Cross-sectional study: minuses
• Cannot determine causality
• - Cannot study rare outcomes
 What if you are interested in
the rare outcome?

The association between regular Red Bull

consumption and…
• A perfect score on the USMLE – Step 2
• Graduating top 1% of the medical school
• class
• Acceptance into a highly selective residency
ANSWER: A Case-Control study
 Study Design #2
A case-control study
• Cases: 4th year med students accepted to
residency in “highly selective specialty X”.
• Controls: 4th year med students who applied but
were not accepted.
• Predictor: self-reported regular Red Bull
consumption
• Additional covariates (age, sex, medical school,
undergraduate institution)
 Case control studies
• Investigator works “backward” (from
outcome to predictor)
• Sample chosen on the basis of outcome
(cases), plus comparison group (controls)
 Case-control study structure
• Red Bull consumption YES
• Red Bull consumption NO

ACTUAL CASES
• 4th year UCSF students
who matched in “highly
selective specialty X”

ACTUAL CONTROLS
4th year students who failed
to match in “highly selective
specialty X”
 Case control studies
• Determines the strength of the association
between each predictor variable and the
presence or absence of disease

• Cannot yield estimates of incidence or

prevalence of disease in the population
(why?)
• Odds Ratio is statistics
 Case-control Study: pluses
• Rare outcome/Long latent period
• Inexpensive and efficient: may be only
feasible option
• Establishes association (Odds ratio)
• Useful for generating hypotheses
(multiple risk factors can be explored)
 Case-control study-minuses
Causality still difficult to establish
• Selection bias (appropriate controls) Caffeine and
Pancreatic cancer in the GI clinic
• Recall bias: sampling (retrospective) Abortion and
risk of breast cancer in Sweden
• Cannot tell about incidence or prevalence
Studies of diagnostic tests:
• Sensitivity, specificity
• Positive predictive value, negative predictive value
 Case-control - “the house red”
Rely tampons and toxic shock syndrome:
• High rates of toxic shock syndrome in menstruating women
• Suspected OCPs or meds for PMS
• Cases: 180 women with TSS in 6 geographic areas
• Controls: 180 female friends of these patients and 180
females in the same telephone code
• Tampon associated with TSS (OR = 29!)
• Super absorbency associated with TSS (OR 1.34 per gm
• increase in absorbency)
• Led to “RELY” brand tampons being taken off the market.
 Where are we?
Preliminary results from our cross-sectional and
case-control study suggest an association
between Red Bull consumption and improved
academic performance among medical students

• What’s missing? - strengthening evidence for a

causal link between Red Bull consumption and
academic performance
• Use results from our previous studies to apply
for funding for a prospective cohort study!
 Study design #3
• Prospective cohort study of UCSF medical
students Class of 2009
• All entering medical students surveyed
regarding beverage consumption and variety of
other potential covariates
• Survey updated annually to record changes in
Red Bull consumption
• Outcomes: USMLE Step 1 score, USMLE Step 2
score, match in first choice residency
 Cohort studies
• A cohort (follow-up, longitudinal) study is a comparative,
observational study in which subjects are grouped by their exposure
status, i.e., whether or not the subject was exposed to a suspected
risk factor
• The subjects, exposed and unexposed to the risk factor, are followed
forward in time to determine if one or more new outcomes
(diseases) occur
Subjects should not have outcome variable on entry
No new subjects allowed in after initial recruitment
• The rates of disease incidence among the exposed and unexposed
groups are determined and compared.
 Elements of a cohort study
Selection of sample from population
Measures predictor variables in sample
Follow population for period of time
Measure outcome variable
Famous cohort studies
• Framingham
• Nurses’ Health Study
• Physicians’ Health Study
• Olmsted County, Minnesota
Prospective cohort study structure
 Strengths of cohort studies
• Know that predictor variable was present
before outcome variable occurred (some
evidence of causality)
• Directly measureincidence of a disease
outcome
• Can study multiple outcomes of a single
exposure (RR is measure of association)
 Weaknesses of cohort studies
Expensive and inefficient for studying rare
outcomes
• HERS vs. WHI
Often need long follow-up period or a very large
population
• CARDIA
Loss to follow-up can affect validity of findings
• Framingham
 Other types of cohort studies
Retrospective cohort
• Identification of cohort, measurement of
predictor variables, follow-up and
measurement of outcomes have all occurred
in the past
• Much less costly than prospective cohorts
• Investigator has minimal control over study
design
 Other types of cohort studies
Nested case-control study
• Case-control study embedded in a cohort study
• Controls are drawn randomly from study sample
Double cohort
• Used to compare two separate cohorts with
different levels of exposure to predictor variable
(e.g., occupational groups)
 What type of study is this?
• Among individuals with coronary disease, what is the
association between baseline levels of B-type natriuretic
peptide and subsequent risk of heart failure?
• Among individuals presenting to heart failure clinic, what
is the association between self-reported symptoms and
risk of hospitalization for heart failure?
Using data from HERS (RCT of HRT in women with coronary disease):
• Determine the risk factors for developing incident heart failure among
women without heart failure at baseline.
• Determine whether HRT is associated with mortality among women
with heart failure.
• Determine genetic markers for development of heart failure among
black women in HERS
 What distinguishes observational
studies from experiments?

Ability to control for confounding

• Predictor
• Outcome
• Confounder
Examples:
• sex (men are more likely to drink red bull and men are
more likely to match in neurosurgery)
• Undergraduate institution (students from northwest
school are more likely to drink red bull and also more
likely to score higher on USMLE)
 But we measured all of the
potential confounders

In a prospective cohort study you can (maybe)

measure all potential known confounders,
but…
You can’t control for unanticipated or
unmeasured confounders
 Study design # 4
Randomized controlled trial of daily Red Bull
consumption among entering UCSF medical
students Class 2009
• Randomized to daily consumption of Red Bull
vs. daily consumption of placebo
• Outcomes: USMLE Step 1 score, USMLE Step
2 score, match in first choice residency
 Randomized controlled trials
• Investigator controls the predictor
variable (intervention or treatment)
• Major advantage over observational
studies is ability to demonstrate causality
• Randomization controls unmeasured
confounding
• Only for mature research questions
 Steps in a randomized
controlled trial

Select participants
• high-risk for outcome (high incidence)
• Likely to benefit and not be harmed
• Likely to adhere
Measure baseline variables
Randomize
• Eliminates baseline confounding
• Types (simple, stratified, block)
 Steps in a randomized
controlled trial

Blinding the intervention

• As important as randomization
Eliminates
• co intervention
• biased outcome ascertainment
• biased measurement of outcome
Follow subjects
• Adherence to protocol
• Lost to follow up
Measure outcome
• Clinically important measures
• Adverse events
 What is Blinding?
• Single blind - participants are not aware of
treatment group
• Double blind - both participants and investigators
unaware
• Triple blind - various meanings
• persons who perform tests
outcome adjudicators
safety monitoring group
 Why blind?: Co interventions
Unintended effective interventions
• participants use other therapy or change
• behavior
• study staff, medical providers, family or
• friends treat participants differently
Nondifferential - decreases power
Differential - causes bias
 Why blind?: Biased Outcome
Ascertainment or adjudication

If group assignment is known

• participants may report symptoms or outcomes
differently
• physicians or investigators may elicit symptoms
or outcomes differently
• Study staff or adjudicators may classify similar
events differently in treatment groups
Problematic with “soft” outcomes
• investigator judgement
• participant reported symptoms, scales
 Analysis of randomized
controlled trial

Analyzed like cohort study with RR

Intention to treat analysis
• Most conservative interpretation
• Include all persons assigned to intervention
group (including those who did not get
treatment or dropped out)
Subgroup analysis
• Groups identified pre-randomization
 High Quality Randomized Trials
• Tamper-proof randomization
• Blinding of participants, study staff, lab staff,
outcome ascertainment and adjudication
• Adherence to study intervention and protocol
• Complete follow-up