If Nothing Goes Wrong,

Is Everything All Right?

Interpreting Zero Numerators
James A. Hanley, PhD, Abby Lippman-Hand, PhD

PHYSICIANS frequently must pro- inferences about the size of a risk, maximum malformation risk compat¬
vide patients with estimates of the and (3) the principles of inferential ible with finding none of 112 infants
risk of a particular medical procedure statistics that apply to nonzero with defects in a single study?
or of the probability of a specific numerators apply equally well to zero Example 3.—The final example is
health outcome. To do so, they may numerators. In fact, there is a quick one we will adapt to explain the
use data from their own experiences and simple rule that establishes the inferences that may be made when a
as well as those available in the maximum long-run risk associated zero numerator is found. In a study of
literature. A general goal is to locate with an observation of no effects in a siblings of 167 infants with tracheo-
the estimate within a fairly narrow sample of any given size. esophageal dysraphism (TED), none

range. was found to have a neural tube

When the procedure or outcome of Examples defect.' (We ignore here problems in
concern is relatively uncommon, pre- Studies reporting a zero numerator defining the denominator for this
cise estimation, although desirable, are fairly common in the literature. study and assume, for illustrative
may be difficult or impossible. More- The following examples are from purposes, that 167 siblings were stud¬
over, when the only data available recent issues of major medical jour¬ ied and that the observed rate of
come from a study in which none of nals, including this one. They have neural tube defect was 0/167.) Is this
the events of concern actually oc- been arbitrarily chosen strictly for evidence sufficient to say that the
curred\p=m-\ie,a study in which a zero illustrative purposes. Thus, we have risk of neural tube defect in siblings
numerator is reported\p=m-\makingin- reduced the details to a minimum. of children with TED is not increased
ferences seems to be particularly For each, the reader should consider over the risk in the general popula¬

problematic. Because the occurrence the inference that might be made tion (1.5 per 1,000 births in the area
of "no events" seems to be viewed as from these data. studied)? Or is a 0/167 rate also
very different both quantitatively and Example 1.—Of 14 boys followed up compatible with a risk that would
qualitatively from the occurrence of for a median of 5'/2 years after make parents of children with TED
one or more events, it is useful to look chemotherapy for leukemia, none had eligible for the prenatal diagnosis of
into someof the statistical and psy- abnormal testicular function (ie, the neural tube defects in subsequent
chological issues that influence the abnormality rate was 0/14).' With pregnancies?
interpretation of a zero numerator. In what risk, if any, of testicular dys¬ Despite differences in focus, the
particular, we would like to empha¬ function might these results be com¬ three examples have the same struc¬
size that (1) a zero numerator does patible? ture: in each study, no adverse ef¬
not necessarily mean "no risk," (2) a Example 2.—The status of 112 live- fects were found—nothing happened.
zero numerator does not preclude born children whose mothers had Only the denominators—the sample
been immunized against rubella was sizes—differ. To illustrate the infer¬
From the Department of Epidemiology and Health studied to assess the risks of gesta¬ ential process employed when there is
(Drs Hanley and Lippman-Hand) and the Centre for
Human Genetics (Dr Lippman-Hand), McGill Univer- tional exposure to the vaccine.2 None a zero numerator, we will use the
sity, Montreal. of the infants born (0/112) had any numbers in the third example and
Reprint requests to Department of Epidemiology consider how these data might be
and Health, McGill University, 3775 University St, congenital malformations associated
Montreal, Quebec, Canada H3A 2B4 (Dr Hanley). with congenital rubella. What is the used to evaluate the risk associated

Downloaded From: http://jama.jamanetwork.com/ by a University of Chicago Libraries User on 08/10/2013

 with a contrast medium to be used by This is the point at which we decide Table 1.—Probability That a
radiologists. (We chose a hypothetical that because a finding that should not Sample of 167 Will Have No
example to avoid controversy about happen (based on what we posit as Complications
the specific details of any reported the truth) seems nevertheless to have
If Long-Run ThenProbability (%) of
study.) happened, it is not just because we Complication Observing Zero
Suppose that the standard contrast are (un)lucky. Rather, since the Rate Is Complications in 167 Is
agent used by radiologists over a long observation really is too surprising if 1 in 10,000 98
period has been shown to cause a the long-run risk we posited were 1 in 1,000 85
1.5 in 1,000 78
serious reaction in about 15 of every correct, we decide that the real risk 1 in 200 43
10,000 patients exposed to it. That is, must be lower. 1 in 100 19
As a matter of convention, a chance 1 in 56 5
the known risk with the old agent is 1 in 25 0.1
1.5 per 1,000 (the same as the popula¬ of 5% has come to be used as a limit
tion risk for neural tube defects in the to credibility; thus, a finding that had
third study cited). Suppose further at least a 5% chance of occurring is Table 2.—Long-Run Risk Ruled Out
that a new contrast agent is intro¬ considered not that surprising ("it by 01 n (With 95% Confidence, ie,
duced. Soon afterward, a report of its can happen"), but anything more 5% Limit of Credibility)
use in 167 patients appears: no extreme is. Table 1 presents a range
of trial values for the long-run risk Rules Out Any
patient has had the reaction of con¬ Long-Run Rate
cern. Can we infer from these data and the chance of observing zero Rate of (%) Higher Than
that, in the long run, the new medium complications in 167 patients associ¬ 0/10 26'(30)t
will carry less risk, more risk, or the ated with each. As Table 1 shows, our 0/20 14(15)
0/30 10(10)
same risk as the old agent? limit for surprise of 5% is reached 0/50 6(6)
We can begin by assuming that the when the true long-run risk is about 0/100 3(3)
0/1,000 0.3(0.3)
long-run risk with the new agent is no two in 100. In other words, the find¬
different from that with the old. ings "fit" or are "not surprisingly 'Derived "exactly" by solving (1—maximum
Under this trial assumption, each different from" any long-run risk of rate)"=0.05
tDerived from rule of three.
patient has a 0.0015 chance of react¬ 2/100 or less. This "plausible range of
ing and, conversely, a 1—0.0015= possibilities for the truth," which maximum risk [ie, 26% if w=10] is
0.9985 chance of not reacting. The extends from zero to the maximum
becoming high that the difference
so
chance that all 167 patients will avoid compatible with our 5% level of sur¬ between it and that suggested by the
a reaction—that the reaction rate prise, is usually termed a "95% confi¬ rule [3/10=30%] is not usually worth
would be 0/167-is 0.9985"7, or 78%. dence interval."
quibbling about.) The derivation of
Thus, finding no reactions among this the "rule of three" is as follows:
group is not at all surprising. In fact,
Making inferences:
if the new agent carries the same risk The Rule of Three To find the largest risk with which the
as the old, it would be more surpris¬ In all of these examples, the event finding of 0/n is compatible (ie, a level of
of interest failed to occur in a finite credibility of at least 5% or 0.05), one must
ing if anyone did react. either proceed by trial and error, as we
If we are more pessimistic and number of subjects. In making infer¬
did, or else explicitly solve the following
believe that the long-run complica¬ ences from observations with zero
equation:
tion rate with this new agent could be numerators, we want to say with a (1-Maximum Risk)"=0.05
much higher than that with the old, certain degree of confidence that the This can be rewritten, by taking the nth
we can assess a different trial value, true or long-run risk (of chemothera¬ root of each side, as
for example, a risk of one in 100. Here py, of rubella immunization, of hav¬ 1-Maximum Risk=^7J7J5 (1)
again, however, observing no reac¬ ing a sibling with NTD) is between The reader who uses a calculator to evalu¬
tions in 167 patients would still not be zero and some upper limit. ate ^WM or 0.05"" for n above 30 will find
that surprising: there would be a To be 95% confident that our inter¬ that it comes remarkably close to (n—3)/w
or 1—(3/??); thus, equation 1 can be rewrit¬
O^g'6', or 19%, chance that none of val estimate of the long-run risk is
ten as follows:
these 167 patients would react to the correct, a simple rule (of unknown 1-Maximum Risk=l-(3/7t)
new agent even if the true risk were origin) can be applied.4 This "rule of or Maximum Risk=3/»
one in 100. three" states that if none of n The reader who would like to under¬
Finally, we can take an even great¬ patients shows the event about which stand "how the 3 got in there" can do so by
trial value, say a risk of four in 100. we are concerned, we can be 95% writing 0.05' as an infinite series, ie,
"

er
Were this the true risk, then there confident that the chance of this 0.051 "=l+[(ln 0.05)/n]
would be just a 0.1% chance of find¬ event is at most three in n (ie, 3/w). In +[(ln 0.05)V2ii-]+ (2)
other words, the upper 95% confi¬ where In 0.05 refers to the natural loga¬
ing no reactions in a group of 167
...

dence limit of a 0/n rate is approxi¬ rithm of 0.05. Given that In 0.05 is -2.9957,
patients, and we would express great or —3 when rounded off to two decimal
surprise at an observation as discrep¬ mately 3/w. (This approximation is places, and given an n of 30 or more, the
ant with the "truth" as this. remarkably good: when n is larger terms involving divisors of ri or bigger in
How high we would go with these than 30, the rule of three agrees with the right-hand side of equation 2 make
trial estimates of the true long-run the exact calculation to the nearest almost no contribution (ie, 0.05' is very
"

risk depends on what might be viewed percentage point; below 30, it slightly close to l-[3/n]).
as our "willingness to be surprised." overestimates the risk, but then the For a 99% confidence interval, the cor-

Downloaded From: http://jama.jamanetwork.com/ by a University of Chicago Libraries User on 08/10/2013

 responding shortcut is a "rule of 4.6" (In a patient, a still-perfect surgical Europe and Russia, reminded
was
0.01=-4.6051...), while for 99.9% confi¬ record, a field trial of a vaccine that circa 1800 that never in the
once
dence one uses a "rule of 6.9." uncovered no major side effects, an preceding 5,000 years had the sun
Some readers will wonder why one can¬ failed to rise—apparently earlier data
not immediately calculate the probability
ophthalmology practice in which no
of 0/w from the Poisson distribution: one
patient with glaucoma was younger were not considered reliable—and
than 23 years, an airline that has was asked to provide odds that it
can indeed do so, solving instead the
never had a fatality. Thus, under¬ would rise on the following day, he
equation exp(—n maximum risk)=0.05.
•

As such readers might expect, the result¬ standing the limits of the inferences suggested odds of roughly n:l, where
ing answer of 3/n is the same. that can be made with such an obser¬ n was the number of days in the

If we apply this rule to our exam¬ vation is important. series, as a fair bet. For someone
It is interesting that although a betting on sunrise who had less trust
ples, and confine our inferences to the
populations from which the patients finding of 0 events in n observations in nature, odds of w/3:l would be an
can be interpreted in much the same even safer gamble.)
derive, we can conclude with 95% statistical way as any other binomial We urge a reformulation of the
confidence as follows:
Example 1.—The maximum risk of rate, it often seems to have a qualita¬ views of a zero numerator and
abnormal testicular function is not tive impact far in excess of its quanti¬ encourage those reporting such obser¬
tative meaning. A few (sometimes vations to consider the maximum risk
greater than 21% (3/w=3/14=21%)
after chemotherapy. Here, the "cor¬ contradictory) clues in the literature with which their findings are compat¬
rect" upper confidence limit based on may help to explain this: (1) People ible. To this end, the confidence inter¬
0/14 is 19%. Interestingly, this "rule tend to ignore the size of the denomi¬ val is helpful since it translates the
of 14 consecutive failures" is com¬ nators on which rates are based.6 For results of a sample not into a single
monly used in cancer treatment instance, a rate of 10% is given much number, but rather into a range that
research5 to screen out agents that the same credence whether it is is quite likely to contain the rate
are unlikely to show activity in at
observed in 20 or 200 cases. Presum¬ characteristic of the population. Be¬
least one patient in five. ably the same holds for the rate of cause a confidence interval may be

Example 2.—The frequency of mal¬ 0%. (2) People tend to focus on constructed easily from a zero numer¬
formations does not exceed 2.7% in numerators. Parents who have had ator using the "rule of three," we
offspring of women immunized genetic counseling, for example, gen¬ hope that those fortunate enough to
against rubella during pregnancy. erally view their risks in binary form: be able to report "no problems so far"
Example 3.—The frequency of neu¬ something either will or will not will quantify the worst or best that a
ral tube defects is not greater than happen, no matter what the actual group of future patients can expect.
1.8% in siblings of patients with risk is.7 A "one" in the numerator
TED. Note that this 0% to 1.8% range never disappears, no matter the size
References
includes the rate of 1.5 per 1,000 in of the denominator. For them, it is
not the odds that matter so much as 1. Blatt J, Poplack DG, Sherins RJ: Testicular
the general population as well as the function in boys after chemotherapy for acute
recurrence risk for second-degree rel¬
the knowledge that the disorder is lymphoblastic leukemia. N Engl J Med 1981;
atives of patients with neural tube possible. Perhaps a zero numerator 304:1121-1124.
2. Preblud SR, Stetler HC, Frank JA, et al:
defects. carries similar weight in that it sug¬ Fetal risk associated with rubella vaccine. JAMA
Table 2 summarizes the rule of gests (falsely) that an event is impos¬ 1981;246:1413-1417.
sible. (3) When one is faced with a 3. Baird P, MacDonald EC: Siblings of chil-
three and tabulates the "true" risks dren with tracheoesophageal dysraphism. Can
that can be ruled out given the occur¬ theoretical risk that has not yet man¬ Med Assoc J 1981;125:1083-1084.
rence of zero events in series of
ifested itself, ie, when something pos¬ 4. Rumke CL: Implications of the statement:
sible has not No side effects were observed. N Engl J Med
different sizes. yet happened, people 1975;292:372-373.
may tend to expect that it cannot 5. Holland JF, Frei E (eds): Cancer Medicine,
Comment happen in the future. Lacking evi¬ ed 2. Philadelphia, Lea & Febiger, 1982, p 535.
6. Tversky A, Kahneman D: Judgment under
Although the examples we have dence that responses may be variable, uncertainty: Heuristics and biases. Science 1974;
discussed reflect our own interests, they then underestimate what the 185:1124-1131.
one may observe a zero numerator in real risk may be. (Interestingly, when 7. Lippman-Hand A, Fraser FC: Genetic coun-
seling: The postcounseling period: I. Parents'
very diverse contexts: a new diagnos¬ the Marquis de Laplace, statistical perceptions of uncertainty. Am J Med Genet
tic test that has not yet misclassified consultant to the gambling royalty of 1979;4:51-71.

Downloaded From: http://jama.jamanetwork.com/ by a University of Chicago Libraries User on 08/10/2013