UNDERSTANDING THE QUALITY OF DATA IN CINICAL MEDICINE | MIDTERMS LESSON 06

UNDERSTANDING THE QUALITY OF DATA IN CLINICAL MEDICINE

OBJECTIVES:

• Define the validity and reliability of screening and diagnostic

tests

• Compare measures of validity, including sensitivity and

specificity

• Introduce positive and negative predictive value

• Address measures of reliability, including percent agreement

and kappa

MEASURES OF VALIDITY

VALIDITY OF SCREENING TESTS

• Validity: Ability of a test to distinguish between WHO HAS a disease and WHO DOES NOT

• Screening test: is performed as a preventative measure – to detect a potential health problem or disease in someone that

doesn't yet have signs or symptoms

• Two components:

o Sensitivity – able to correctly identify who has the disease

o Specificity – correctly identifies who does not have the disease

TESTS WITH DICHOTOMOUS RESULTS

• How about tests with continuous variables?

o Such as blood pressure or blood glucose levels

o There is no “positive” or “negative” result

SAN PEDRO COLLEGE | Medical Laboratory Science Department Page 1 of 17

UNDERSTANDING THE QUALITY OF DATA IN CINICAL MEDICINE | MIDTERMS LESSON 06

o Cutoff point

• Different cut-points yield different sensitivities and specificities

• The cut-point determines how many subjects will be considered as having the disease

• The cut-point that identifies more true negatives will also identify more false negatives

• The cut-point that identifies more true positives will also identify more false positives

• The choice of a high or a low cut off level for screening therefore depends on the importance we attach to false positives
and false negatives

USE OF MULTIPLE TESTS

SAN PEDRO COLLEGE | Medical Laboratory Science Department Page 2 of 17

UNDERSTANDING THE QUALITY OF DATA IN CINICAL MEDICINE | MIDTERMS LESSON 06

• Simultaneous testing- conduct at the same time

• Sequential (two-stage) testing

SIMULTANEOUS TESTING

• When two or more tests are conducted in parallel

• Disease positives are identified as those who test positive by either one test or by both tests

• Goal: maximize probability that subjects with the disease (true positives) are identified

• Consequently: more false positives are also identified

NET SENSITIVITY USING TWO SIMULTANEOUS TESTING

STEPS:

1. Calculate Test A sensitivity

2. Calculate Test B sensitivity

3. Take the number of positive people identified by both tests by multiplying Test B sensitivity to the positive cases of Test A

4. Calculate the portion of positives identified by Test A only by using this number and subtracting from Test A positive cases,
place in numerator

5. Calculate the portion of positives identified by Test B only by using this number and subtracting from Test B positive cases,
place in numerator

6. Take the net sensitivity by dividing all of these over total number of cases with disease, expressed as a percentage

SAN PEDRO COLLEGE | Medical Laboratory Science Department Page 3 of 17

UNDERSTANDING THE QUALITY OF DATA IN CINICAL MEDICINE | MIDTERMS LESSON 06

160*0.90= 144

NET SPECIFICITY USING TWO SIMULTANEOUS TESTING

STEPS:

1. Calculate Test A specificity

2. Calculate Test B specificity

3. To be called negative in simultaneous tests, only people who test negative on both tests are considered to have true
negative results.

4. Calculate this by multiplying Test B specificity to number of negative cases in Test A

5. Calculate net specificity by dividing this number over the total number of people without the disease, express in percentage

SAN PEDRO COLLEGE | Medical Laboratory Science Department Page 4 of 17

UNDERSTANDING THE QUALITY OF DATA IN CINICAL MEDICINE | MIDTERMS LESSON 06

480*.90(higher specificity) = 432

SEQUENTIAL (TWO-STAGE) TESTING

• After the first (screening) test was conducted, those who tested positive were brought back for the second test to further
reduce false positives

• First test: less expensive, less invasive or less uncomfortable

• Second test: more expensive, more invasive or more uncomfortable

• Consequently: the overall process will increase specificity but with reduced sensitivity

SAN PEDRO COLLEGE | Medical Laboratory Science Department Page 5 of 17

UNDERSTANDING THE QUALITY OF DATA IN CINICAL MEDICINE | MIDTERMS LESSON 06

Prevalence of 5%= 10,000*.05=500 w diabetes net sensitivity= 315/orig pop (500)(1 st test)

True +: 500*.7(sensitivity) net spec= 9,310/orig pop (9,500)= 98%

True -: 9,500*.8 gain in net specificity

COMPARISON BETWEEN TESTS

• Sequential testing:

o Net loss in sensitivity

o Net gain in specificity

• Simultaneous testing:

o Net gain in sensitivity

o Net loss in specificity

• Depends on:

o Aim of testing (screening or diagnosing)

o Costs

o Degree of invasiveness

o Practical considerations

o Third party insurance

GOALS OF DATA COLLECTION AND ANALYSIS

• Clinical medicine requires constant collection, evaluation, analysis, and use of quantitative and qualitative data
• Data is used for diagnosis, prognosis, and choosing and evaluating treatments
• Error:
o Mistakes in the diagnosis and treatment of patients

SAN PEDRO COLLEGE | Medical Laboratory Science Department Page 6 of 17

UNDERSTANDING THE QUALITY OF DATA IN CINICAL MEDICINE | MIDTERMS LESSON 06

o Mistakes due to clear negligence

• Limitations of medical histories, results, and reports
• Goal: minimize error in data so as to guide, not mislead
o Promoting accuracy and precision
o Reducing differential and nondifferential errors
o Reducing intraobserver and interobserver variability

PROMOTING ACCURACY AND PRECISION

• Accuracy
o Close to the true value
o Ability of a measurement to be correct on the average
• Precision
o Also known as reproducibility or reliability
o Ability of a test to give the same result or a similar result with repeated measurement of the same factor

REDUCING DIFFERENTIAL AND NONDIFFERENTIAL ERRORS

• Differential error: happens when the information errors differ between groups

• Nondifferential error: happens when the information is incorrect, but is the same across groups

DIFFERENTIAL ERRORS

• Measurement bias: refers to any systematic error that may occur during the collection of baseline or follow-up data
• Examples:
o Blood pressure values
o measuring height with shoes on
o laboratories and the use of different methods

SAN PEDRO COLLEGE | Medical Laboratory Science Department Page 7 of 17

UNDERSTANDING THE QUALITY OF DATA IN CINICAL MEDICINE | MIDTERMS LESSON 06

NONDIFFERENTIAL ERRORS

• Random error
o Variability and unpredictability
o results in lack of precision
o When data have only random errors, some observations are too high and some are too low

REDUCING INTRAOBSERVER AND INTEROBSERVER VARIABILITY

• Intraobserver variability (within the observer)

o Amount of variation between the results obtained by one observer examining the same results more than once
o Example: same clinician takes successive blood pressure on the same patient

• Interobserver variability (between observers)

o Amount of variation between the results obtained by 2 or more observers examining the same material
o Example: x-ray reading (iba ibang observation) 2 nd opinion

APPLICATION OF VALIDITY MEASURES IN DIAGNOSTIC AND SCREENING TESTS

A. False-positive and false-negative results

B. Sensitivity and Specificity

C. Predictive values

D. Likelihood ratios

FALSE-POSITIVE AND FALSE-NEGATIVE RESULT

 Type 1 error / false-positive error / alpha error

o If something is said to be true when it is actually false
o False positive results: Finding a positive result in a patient in
whom disease is absent

 Type 2 error / false-negative error / beta error

o If something is said to be false when it is actually true
o False negative result: Finding a negative result in a patient whom disease is present

SAN PEDRO COLLEGE | Medical Laboratory Science Department Page 8 of 17

UNDERSTANDING THE QUALITY OF DATA IN CINICAL MEDICINE | MIDTERMS LESSON 06

Hypothesis Testing:

 Statement of a null hypothesis and the selection of a level of significance

 False-positive result (Type 1 error): Happens when we reject a null hypothesis that is actually true

 False-negative result (Type 2 error): Happens when we fail to reject a false null hypothesis

 Sensitivity

o Reliably finding a disease when it is present

o Avoids false negative results

 Specificity

o Reliably excluding a disease when it is absent

o Avoids false positive results

• The stage of the disease often influences the test results

• Very early in the course of almost any infection, a patient may have no immunologic evidence of infection, and tests done
during this time may yield false-negative results

• False-negative results may also occur late in infections such as tuberculosis, when the disease is severe and the immune
system is overwhelmed and unable to produce a positive skin test result (anergy)

SAN PEDRO COLLEGE | Medical Laboratory Science Department Page 9 of 17

UNDERSTANDING THE QUALITY OF DATA IN CINICAL MEDICINE | MIDTERMS LESSON 06

PREDICTIVE VALUES

 Describe the probability of having actual disease given the results of a test

 Positive predictive value (PPV)

o Indicates what proportion of the subjects with positive test results actually have the disease

o
 Negative predictive value (NPV)
o Indicates what proportion of the subjects with negative test results actually do not have the disease

SAN PEDRO COLLEGE | Medical Laboratory Science Department Page 10 of 17

UNDERSTANDING THE QUALITY OF DATA IN CINICAL MEDICINE | MIDTERMS LESSON 06

• Sensitivity and specificity are characteristics of a test.

• Positive predictive value (PPV) and negative predictive value (NPV) are best thought of as the clinical relevance of a test.

• Predictive values are greatly influenced by disease prevalence

• The higher the prevalence, the higher the predictive value

Why should we be concerned about the relationship between predictive value and disease prevalence?

LIKELIHOOD RATIOS

LIKELIHOOD RATIO POSITIVE (LR+)

• “RULE IN DISEASE”

Ratio of the sensitivity of a test to the false-positive error rate of the test

Equation: [a/(a+c)] / [b/(b+d)] or sensitivity / (1−specificity)

Interpretation: ratio must be larger than 1 (test is good)

SAN PEDRO COLLEGE | Medical Laboratory Science Department Page 11 of 17

UNDERSTANDING THE QUALITY OF DATA IN CINICAL MEDICINE | MIDTERMS LESSON 06

LIKELIHOOD RATIO NEGATIVE (LR-)

• “RULE OUT DISEASE”

Ratio of the false-negative error rate divided by the specificity

Equation: [c/(a+c)] / [d/(b+d)] or (1-sensitivity) / specificity

Interpretation: the smaller the LR-, the better the test is

• Sensitivity and specificity assess how good a test is at diagnosis of disease while likelihood ratio predicts the risk of disease
for a particular test.

• Predictive value and likelihood both predict risk of disease, but, unlike predictive value which depends on the prevalence of
the disease, likelihood ratio does not.

QUESTIONS:

Which test would be best to RULE IN infection? LR+ procalcitonin

Which test would be best to RULE OUT infection? LR- C reactive protein

NOTE: If the LR positive of a test is large and the LR negative is small, it is probably a good test

• Ratio of LR+ and LR -

Obtain a measure of separation between the positive and the negative test

Interpretation: <50 = weak test

RECEIVER OPERATING CHARACTERISTIC CURVE

• Cutoff point

Can be determined by likelihood ratios

The limit at which something is no longer applicable / possible

SAN PEDRO COLLEGE | Medical Laboratory Science Department Page 12 of 17

UNDERSTANDING THE QUALITY OF DATA IN CINICAL MEDICINE | MIDTERMS LESSON 06

• Receiver operating characteristic (ROC) curve

o Method used to decide on a good cutoff point
o can be considered a graph of the LR+

• Best cutoff point: Point closest to the upper left corner representing a sensitivity of 100% + false positive error rate of 0%

Area under the curve

• Method of comparing tests

• use a statistical test of significance to decide if the area under one curve
differs significantly from the area under the other curve

• The greater the area under the curve, the better the test is

MEASURES OF RELIABILITY

• Overall percent agreement

• Paired observation

• Multiple variables

• Kappa test ratio

OVERALL PERCENT AGREEMENT (OPA)

• Common way to measure agreement

• Interpretation: If 90% of the observations are in cells a and d = OPA 90%

• Drawbacks:

1. Does not indicate prevalence

2. Does not show how disagreement occurred

SAN PEDRO COLLEGE | Medical Laboratory Science Department Page 13 of 17

UNDERSTANDING THE QUALITY OF DATA IN CINICAL MEDICINE | MIDTERMS LESSON 06

3. Agreement might be due to chance alone

PERCENT AGREEMENT (PAIRED OBSERVATIONS)

OVERALL PERCENT AGREEMENT (MULTIPLE VARIABLES)

KAPPA TEST RATIO

• Measures the extent to which agreement exceeds that expected by chance

• How much better is the agreement between two observers’ readings than would be expected by chance alone?

• What is the most that the two observers could have improved their agreement over the agreement that would be expected
by chance alone?

SAN PEDRO COLLEGE | Medical Laboratory Science Department Page 14 of 17

UNDERSTANDING THE QUALITY OF DATA IN CINICAL MEDICINE | MIDTERMS LESSON 06

Inference: Pathologist A and B have an almost perfect agreement on the results of the histologic classification by subtype of the
75 slides of non-small cell carcinoma with a kappa statistic of 0.81

SAN PEDRO COLLEGE | Medical Laboratory Science Department Page 15 of 17

UNDERSTANDING THE QUALITY OF DATA IN CINICAL MEDICINE | MIDTERMS LESSON 06

1. All the computations in this topic are expressed in percentage with the exception of the

likelihood ratio and kappa test ratio. Meaning to say, there are no other multiplication
factors here but 100 and that the word "per 100" no longer applies.

SAN PEDRO COLLEGE | Medical Laboratory Science Department Page 16 of 17

UNDERSTANDING THE QUALITY OF DATA IN CINICAL MEDICINE | MIDTERMS LESSON 06

a/(a+c) or TP/(TP+FN)

SAN PEDRO COLLEGE | Medical Laboratory Science Department Page 17 of 17