Base Rate Fallacy and COVID-19 Testing - Considerations For Policy

otherwise it can result in seriously mistaken
Base Rate Fallacy and decisions. Likewise, it is also important to try to

communicate this, curious yet possibly
COVID-19 Testing – consequential, piece of mathematical reasoning
Considerations for Policy to the widest section of citizens.
Mainak Seni As the pandemic spreads the prevalence rate
May 2020
starts at very low values and then increases. A
A classic application of one of the great successful control strategy will mean that the
theorems of probabilityii is the so called base- prevalence remains low (less than 1% of the
rate fallacy – a result that common people find population). But, this will also mean that the
counterintuitiveiii, but which is well known to majority of people identified as positive by any
most practitioners and students of statistics. less-than-perfectix test will be actually free of
When cast in the context of epidemiology it the virus. One needs to very carefully evaluate
takes the form: the health and economic costs, both to the
individual concerned and to the society as a
For a rare disease, many (often, the majority) of whole, of isolation / quarantine of merely “test-
the positive results of diagnostic tests, even positive” individuals, in the absence of either
accurateiv ones, are actually false positives. clinical symptoms of disease or strong index of
suspicion.
For example, if a disease is actually present in
only 1 out of every 1000 people in the Moreover, if most “test-positive” individuals are
populationv, then even with a “99.9% accurate asymptomatic, to infer from there that most
test”, half of the people whose test results say carriers of the virus are asymptomatic will be
that they have the disease actually do not. More fraught with risk of error; it is most likely they
surprisingly, if the test is even slightly less are not infected at all (i.e., are false positives)
accurate, 99%, then the proportion of false and, hence, asymptomatic.
positives out of all test-positives (the False
Discovery Rate, FDRvi) is more than 90%; and Decision making is further complicated, thus
with 98% accuracy, it is over 95%! Fig 1 shows requiring even more careful consideration, by
the nature of the relationshipvii between the FDR the facts that a) the exact prevalence rate is
and the prevalence rate when the prevalence is unknown, b) the prevalence rate may be widely
low. different across sub-populationsx, and c) it may
increase rapidly with time, particularly if control
The only “escape” from this fallacy is a perfect strategies are ineffectivexi, causing markedly
test (“100% accurate”)viii that, like all perfect decreased FDR.
things, is hard to come by.
Somewhat assuringly (perhaps) the results are

closer to common intuition if the disease is
more prevalent. For example, if 1 in 5 people in
the population have the disease, then the
proportion of false positives even with only the
“95% accurate” test is a little more than one-
sixth. And, it reduces to the intuitive 1 in 20
when the prevalence is 1 out of 2.
Fig 1: FDR vs Prevalence for “rare” event
For an infectious, and potentially rapidly
growing, pandemic like COVID-19, policy
makers need to be aware of this fallacy as
page 1 of 3
i Director and Head of Analytics, Business Brio, Kolkata
ii Bayes’ Theorem. Although the formal probability statement is a textbook application of this famous theorem, the
informal sample proportion statement (like how it is stated here) can easily be derived from first principles by
elementary arithmetic; see vii below
iii This is a case of a type of bias in judgment that, some cognitive psychologists have argued, arises out of
insensitivity to prior probability of outcomes. Refer to page 1 of:
Judgment under Uncertainty: Heuristics and Biases
Author(s): Amos Tversky and Daniel Kahneman
Source: Science, New Series, Vol. 185, No. 4157, (Sep. 27, 1974), pp. 1124-1131
https://www.jstor.org/stable/1738360?origin=JSTOR-pdf
iv Accuracy may be misleading and so usually specificity and sensitivity are considered separately; here “high
accuracy” is meant as simultaneously high specificity and sensitivity. In this paper, for simplicity “99% accuracy”
denotes specificity = sensitivity = 99%, etc., though in reality specificity and sensitivity of a test are usually not the
same. For low prevalence, however, it is the specificity that is more influential on FDR, though
v To put this in the context of COVID-19, as of 21 May 2020, Qatar has among the highest case prevalence as a % of
the whole population at 1.34%; on that same date the rates for the US and India are 0.49% and 0.0086%
respectively [Source: https://www.worldometers.info/coronavirus/]
vi False Discovery Rate = Prob(Actual Infected = 0 | Test Result = 1)

Actual Infected status must be judged independently by some “Gold Standard” criteria (may be, another more
sensitive test)
from ibid v: Rounding up India’s prevalence rate to 0.01%, and with a test accuracy of 99.9%, the FDR is more
than 90%
Table: Computed FDR for Some Countries
Test “Accuracy”
Prevalence 99.90% 99.00% 98.00%
Qatar 1.34% 6.90% 42.70% 60.04%
US 0.49% 16.90% 67.20% 80.6
India 0.0086% 92.10% 99.10% 99.60%
It should be noted that these quoted “case prevalence rates” are simply computed by dividing the number of
positive cases (usually from tests) with the country’s population and, the testing strategy and coverage vary among
countries and with time
vii
( 1−specificty )×( 1− prevalence )
FDR=
sensitivity× prevalence+ ( 1−specificty )×( 1− prevalence )
1
=
sensitivity prevalence
1+ ×
1− prevalence 1−specificity
1
=
sensitivity 1
1+ ×
1−specificity 1
−1
prevalence
The total cases that are identified as positive (the denominator in the first expression above) is the sum of two
components: a) those that are actually positive and correctly identified as such, and b) those that are actually
negative but erroneously identified as positive.
Low prevalence means that there are many more negative cases than positive ones in the population. Specificity is
the measure of how well the test can correctly identify the real negative cases. A test with slightly imperfect
specificity will have a small false positive rate (as a % of real negatives) but since the number of real negatives are
very high, this will still translate to a significant number of false positives.
Sensitivity, on the other hand, measures how well the test can correctly identify the real positive cases. Now,
because of low prevalence, the number of real positive cases will be small to begin with. This not only makes
component b) dominant over component a) and hence results in high FDR (by definition) but also makes the FDR
sensitive to small changes in the specificity, as the high number of real negatives accentuates the small false
positive rate. The fallacy becomes pronounced whenever the false positive rate is larger than the (low) prevalence
rate. Also, ibid iv, vi
viii Strictly speaking a test with perfect (= 100%) specificity irrespective of (even poor, but non-zero) sensitivity. It is,
however, not easy to empirically establish perfect specificity (to say nothing of the technical difficulties of
developing such a test in the first place): Even if 100 known disease free individuals are all correctly identified by
the test as negative then also the 95% confidence interval for the specificity is (96.4%, 100%)
For example of a real time RT-PCR for SARS-CoV-2 test specificity see
https://www.fda.gov/media/136231/download
ix Less-than-perfect in the sense of specificity < 100%
x It is crucial to consider the prevalence rate in the sub-population that is being tested; If only symptomatic and/or
high-exposure-risk (e.g., medical providers or close contacts of otherwise confirmed cases) people are being tested
then the prevalence rate among them might be a lot higher than what would be the case if the general population are
being tested at random. Or, conversely, if the general population, regardless of symptoms and index of suspicion,
are being tested at random, then there will be much higher FDR; this can be a reasonable argument for not doing
general testing in the early stages of the pandemic
xi Or prevalence may decrease if testing policy changes from symptomatic-and-high-risk-only-testing to more

inclusive general testing causing the FDR to increase

Base Rate Fallacy and COVID-19 Testing - Considerations For Policy

Uploaded by

Copyright:

Available Formats

Base Rate Fallacy and COVID-19 Testing - Considerations For Policy

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Base Rate Fallacy and COVID-19 Testing - Considerations For Policy

Uploaded by

Copyright:

Available Formats

otherwise it can result in seriously mistaken

Base Rate Fallacy and decisions. Likewise, it is also important to try to

Somewhat assuringly (perhaps) the results are

vi False Discovery Rate = Prob(Actual Infected = 0 | Test Result = 1)

ix Less-than-perfect in the sense of specificity < 100%

xi Or prevalence may decrease if testing policy changes from symptomatic-and-high-risk-only-testing to more

You might also like