MEDICINE

REVIEW ARTICLE

Concordance Analysis
Part 16 of a Series on Evaluation of Scientific Publications

Robert Kwiecien, Annette Kopp-Schneider, Maria Blettner

any diagnostic methods in medicine carry the

SUMMARY
Background: In this article, we describe qualitative and
M risk of misdiagnosis. All physicians sometimes
make diagnostic errors, any two physicians will some-
quantitative methods for assessing the degree of agree-
times disagree on a diagnosis, and even the technical
ment (concordance) between two measuring or rating
measurements on which diagnoses are based are never
techniques. An assessment of concordance is particularly
perfectly accurate. A common feature of medical diag-
important when a new measuring technique is introduced.
nosis and technical measurement is that both of them,
Methods: We give an example to illustrate a number of as a rule, are erroneous, or at least susceptible to error.
simple methods of comparing different measuring or rat- In this article, we will refer to persons making diag-
ing techniques, and we explain the underlying principle of noses, as well as to diagnostic methods and measuring
each method. We also give further illustrative examples techniques, as raters, and to the diagnoses and
from medical research papers that were retrieved by a measurements that they make as ratings.
selective literature search. If some technique exists with which the quantity of
Results: Methods of comparing different measuring or rat- interest can actually be measured without error, this
ing techniques are of two kinds: those with a nominal rat- technique is called a gold standard. Now, suppose
ing scale and those with a continuous rating scale. We that a new technique is to be introduced for measur-
only discuss methods for comparing one measuring or ing tumor volume (for example) more readily, or with
rating technique with another one. Moreover, we point out less trouble for the patient, than with the established
some common erroneous approaches to concordance technique (or gold standard). We will then want to
analysis. know how well the measurements obtained with the
Conclusion: Concordance analysis is needed to establish new technique agree with those obtained by the old
the validity of a new diagnostic measuring or rating tech- one. A common but incorrect method of comparing
nique or to demonstrate the near-equivalence of multiple two measuring techniques for a quantity on a con-
measuring or rating techniques. Erroneous approaches to tinuous scale (e.g., tumor volume) is to calculate a
concordance analysis can lead to false conclusions. correlation coefficient between two sets of measure-
ments obtained by the two techniques.
Cite this as: We will explain why the correlation coefficient is
Kwiecien R, Kopp-Schneider A, Blettner M: Concordance an unsuitable indicator of the degree of agreement
analysispart 16 of a series on evaluation of scientific
(concordance) between two quantitative measuring
publications. Dtsch Arztebl Int 2011; 108(30): 51521.
techniques. Rather, the results obtained by them
DOI: 10.3238/arztebl.2011.0515
should be displayed graphically, as we will demon-
strate below, so that the physician can directly assess
the quality of agreement.
Measuring the agreement of a new technique with a
gold standard includes determining the measuring
error of the new technique. In principle, however, this
is done in precisely the same way as determining the
degree of agreement of two measuring techniques that
are both susceptible to error.
Sometimes, one would like to assess the agreement
between two raters of a nominal variable (e.g., in-
Institut fr Biometrie und Klinische Forschung (IBKF), Westfhlische Wilhelms- fluenza, flu-like illness, or other), or of a vari-
Universitt Mnster: Dr. rer. nat. Kwiecien able that is both nominal and ordinal (e.g., good,
Abteilung Biostatistik, Deutsches Krebsforschungszentrum (DKFZ), Heidelberg: fair, or poor). For instance, one might like to
Prof. Dr. rer. nat. Kopp-Schneider
know how closely two high-school teachers agree on
Institut fr Medizinische Biometrie, Epidemiologie und Informatik (IMBEI),
Universittsmedizin der Johannes Gutenberg Universitt Mainz: gradings of term papers, or how closely two doctors
Prof. Dr. rer. nat. Blettner agree when diagnosing patients as healthy or ill.

Deutsches rzteblatt International | Dtsch Arztebl Int 2011; 108(30): 51521 515
MEDICINE

FIGURE 1 along a continuous scale to each member of the

sample.

Ratings on a continuous scale

Most physical measurements are on a continuous
numerical scale. Often, there is more than one tech-
nique or instrument for measuring the quantity in
question, and the question arises how closely these
techniques agree (1). If one wishes to introduce a
new method of measuring a medical variable, one
must first evaluate its validity by checking how well
it agrees with an already established method, or with
a gold standard.
In this section, we will present statistical methods
for comparing two measuring techniques and apply
them to some fictitious examples. We assume that
some number n of persons or objects (perhaps 100 of
them) undergo measurement with each of the two
techniques, yielding a total of n pairs of measure-
ments. As a first step, the measurements obtained by
the two techniques are plotted against each other in a
graph: one point is plotted for each member of the
sample, its x-coordinate being the measurement ob-
tained by the first technique and its y-coordinate the
measurement obtained by the second technique. If
the two techniques agree perfectly or nearly so, then
all the plotted points should lie on or near the diago-
nal line x = y.
Two distinct and readily understandable situations
are shown in Figures 1a and 1b (Examples a and b).
Any pair of measurements that were precisely equal
(Measurement 1 = Measurement 2) would be plotted
as a point lying on the diagonal line x = y, which is
drawn on both graphs. In Example a, the two measur-
ing techniques agree closely; in Example b, however,
the plot at once reveals that the difference between
Measurements 1 and 2 varies ever more widely for
increasing values and is greater overall than in
Direct comparison of two raters with a point cloud and the
Example a.
diagonal line x = y; Measurement 1 vs Measurement 2 in the two A more informative way of displaying such rela-
examples discussed in the text: Example a above, Example b below tionships is the so-called Bland-Altman diagram,
shown for the two Examples in Figures 2a and 2b. As
before, each pair of measurements is plotted in the
x-y plane, but in a different way: The average of the
Our present concern is not whether the raters give two measurements is plotted as the x-coordinate, and
correct ratings, but rather how closely they agree. the difference between them as the y-coordinate. In
The situation becomes more complicated if we want addition, the mean of all differences is plotted as a
to know how closely more than two raters agree with solid horizontal line, and two additional (dotted)
one another; we will not discuss this any further here. horizontal lines are plotted above and below this line
We will present descriptive methods of evaluating at a distance of 1.96 times the standard deviation of
interrater agreement visually and quantitatively. Such the differences. These two lines correspond to the
methods constitute what is called concordance analy- so-called limits of agreement. The mean-of-all-
sis. Our discussion will center on Bland-Altman differences line indicates a systematic deviation of
diagrams and Cohens kappa. We will deal with two the two measuring techniques for which, in general, a
different situations: In one situation, two raters correction can be introduced; the limits of agreement
assign a nominal rating, such as healthy or ill indicate the size of further deviations that in general,
(dichotomous) or influenza, flu-like illness, or are not correctable. If the quantity being measured is
other (more than two alternatives), to the n members normally distributed, then 5% of the measured differ-
of a sample of persons or objects to be rated. In the ences ought to lie beyond the limits of agreement,
other situation, two raters assign a numerical quantity i.e., more than 1.96 standard deviations above or

516 Deutsches rzteblatt International | Dtsch Arztebl Int 2011; 108(30): 51521
 MEDICINE

FIGURE 2 FIGURE 3

Comparison of two raters with a Bland-Altman diagram; Point cloud diagrams for comparing two functionally related
diagrams are shown for Example a (above) and Example b (below) measuring techniques; Measurement 1 vs Measurement 2 for
Example c (above) and Example d (below)

below the mean of all differences (2). The factor 2 is cases; this difference is small in relation to the
often used, for simplicity, instead of 1.96; the latter, measured quantities themselves. The distance be-
however, corresponds more precisely to the 97.5% tween the two limits of agreement (in other words,
quantile of the normal distribution. In summary, the the width of the region of agreement) is 0.2 in this
Bland-Altman diagram is a useful aid that enables a example.
visual comparison of measuring techniques. When Bland-Altman diagrams are used in real-life
In Figure 2a, the Bland-Altman diagram for situations to see how well two measuring techniques
Example a confirms that the two measuring tech- agree, the question whether the observed degree of
niques are in close agreement. The mean-of-all- agreement is good enough can only be answered in
differences line is very near 0; thus, there seems to be relation to the particular application for which the
no systematic deviation between the measured values techniques are to be used (i.e., good enough for
of the two techniques. In this example, the standard what?). Prospective users must decide how closely
deviation of all differences is roughly 0.05. Assuming the measurements must agree (otherwise stated: how
that the quantity being measured is normally dis- narrow the band between the limits of agreement
tributed, we can conclude that the difference between must be) to be acceptable for clinical purposes.
the two measurements will be less than 0.1 in 95% of Tetzlaff et al. (1), for instance, compared magnetic

Deutsches rzteblatt International | Dtsch Arztebl Int 2011; 108(30): 51521 517
MEDICINE

BOX 1 resonance imaging (MRI) with spirometry for a

specific clinical application using Bland-Altman dia-
grams (among other methods) and found the degree
Calculating Cohens kappa: an illustrative example of agreement to be satisfactory.
The Bland-Altman diagram for Example b (Figure
2b) immediately reveals more than one limitation to
Rater 2 the agreement of the two measuring techniques being
investigated. The mean difference between the two
Rater 1 healthy ill row totals measurements is once again near zero, but the limits
of agreement are 1.4 units above and below the mean
healthy 50 (0.45) 10 (0.09) 60 (0.54)
value, i.e., one can expect 95% of all measured
ill 30 (0.27) 20 (0.18) 50 (0.45) differences to lie in the range 1.4 to +1.4. The
physician must decide whether a deviation of this
column totals 80 (0.73) 30 (0.27) 110 magnitude is acceptable. Moreover, the non-uniform
distribution of the points in this diagram indicates
systematic distortion (systematic bias).
Even so, however, poor agreement in a Bland-
Let us suppose that two doctors (Raters 1 and 2) examine 110 patients for
Altman diagram should not lead us to reject a new
the presence of a particular disease and then state whether each patient is
measuring technique prematurely. In Figure 3, two
healthy or ill. Suppose further that Raters 1 and 2 arrive at the same diagno-
further cases (Examples c and d) are shown in which
sis in 70 of 110 patients. The above contingency table contains all of the
the two measuring techniques obviously do not agree
relevant data on the absolute and relative frequencies of agreement and dis-
(the plotted points lie far away from the line of agree-
agreement in our fictitious example.
ment), yet they are nonetheless functionally related,
Raters 1 and 2 agreed on the diagnosis healthy in 45% of cases, and
as the regression curve shows in each case. The
they agreed on the diagnosis ill in 18% of cases. Their probability of agree-
relation between the two techniques is linear in
ment is thus p0 = 70/110 = 45% + 18% = 63%. Yet, even if one rater (or both)
Example c (Figure 3c), nonlinear in Example d
were assigning diagnoses at random, the two of them would sometimes
(Figure 3d).
agree. The expected probability of agreement if this were so can be calculat-
Thus, it often happens that one measurement can
ed from the marginal frequencies in the contingency table (which are given in
be accurately predicted from the other one because
the boxes marked row totals and column totals). Mathematically speaking,
the two of them are clearly functionally related, even
this is the situation called stochastic independence: One raters judgment
though the two measurements themselves yield very
contains no information at all about the other raters judgment.
different values. In Figure 3d, for example, when
For clarity in the following discussion, we will not always represent frac- Measurement 1 yields the value 3.0, we can use the
tions and probabilities as percentages, but will sometimes write them as regression curve to estimate that Measurement 2 will
numbers between 0 and 1 instead: for example, 0.54, rather than 54%. yield the value 7.65. The apparent lack of agreement
Now, if one rater were assigning diagnoses at random, we would expect between the two measuring techniques is thus largely
agreement on the diagnosis healthy with probability 0.54 0.73 = 0.39, and correctable. Having corrected Measurement 2 in
agreement on the diagnosis ill with probability 0.45 0.27 = 0.12. The over- this way by means of the regression curvewhich
all probability of agreement if one rater assigns diagnoses at random is thus corresponds to our best estimate of the functional re-
pe = 0.54 0.73 + 0.45 0.27 = 0.52 (57.2 of 110 cases). In other words, lation between the two measurementswe can com-
there would be agreement in 52% of cases, rather than 63%, as was actually pare the corrected Measurement 2 with Measurement
observed. The observed probability of agreement exceeds the probability 1 using the methods already described, e.g., a new
that would have been expected from random diagnosis by the amount p0 pe Bland-Altman diagram. This procedure closely re-
= 63% 52% = 11%. sembles the calibration of a measuring instrument.
We now norm the excess frequency of agreement over chance in order The determination of the functional relation itself,
to obtain a quantity that cannot be higher than 1. We do so by dividing the i.e., the generation of regression curves of the types
value p0 pe , whatever it may be, by the highest value it can theoretically seen in Figure 3, requires a variety of statistical
have, which is 1 pe . In our example, 1 pe = 100% 52%. This theoretical methods, such as linear and nonlinear regression, that
highest value of p0 pe corresponds to the case where the raters agree 100% we cannot discuss here in any further detail.
of the time. The normed value 2 = (p0 pe)/(1 pe) is called Cohens kappa. The Pearson correlation coefficient (2) between
In our example, Cohens kappa has the value 11% / (100% 52%) = 0.23. the two measuring techniques is often considered to
Cohens kappa equals 1 when the two raters agree in every case; it is 0 demonstrate a linear relationship (thus, a specific
when they agree just as frequently as would have been expected if one rater kind of functional relationship) between them. In-
(or both) were assigning ratings at random. Cohens kappa hardly ever takes deed, a coefficient with a high absolute value (near 1
on its theoretical minimum value of 1. or 1) does indicate such a relationship. A common
error, however, is to misinterpret the implications of
significance tests that are applied to correlation co-
efficients. A finding that the correlation between two
measuring techniques differs significantly from zero
does not necessarily indicate that the two techniques

518 Deutsches rzteblatt International | Dtsch Arztebl Int 2011; 108(30): 51521
 MEDICINE

are in good agreement. Even the slightest, practically TABLE 1

irrelevant relationship between two techniques could,
in principle, yield a statistically significant finding of Categorization of values of Cohens kappa (2)
this type. A significant correlation actually con- Value of k Quality of agreement
tains no information at all about the size of the dis-
agreement between the two types of measurement (3, 4). <0.20 Poor
0.210.40 Fair
Ratings on a nominal scale: Cohens kappa
We now turn to the topic of ratings on a nominal 0.410.60 Moderate
scale. In medical research, the degree of agreement 0.610.80 Good
between two raters is often assessed with a measure 0.811.0 Very good
called Cohens kappa. Song et al. (5), for example,
compared two methods of detecting bone metastases,
which turned out to agree well, with a kappa value of
0.732. What Cohens kappa measures, concisely
stated, is the normed difference between the rate of BOX 2
agreement that is actually observed and the rate of
agreement that would be expected purely by chance. Cohens kappa for scales with k categories (k 2)
What does this mean in concrete terms? We will
use a fictitious example to illustrate the use of The contingency table below applies to the more general problem of comparing
Cohens kappa for a dichotomous rating. The two raters who use a rating scale with an arbitrary number of categoriesnot
procedure for calculating Cohens kappa is described necessarily two, as in our discussion up to this point. The row and column totals
in greater detail in Box 1. Let us assume that two doc- are ai. = ai1 ++ aik und a.i = a1i ++ aki , respectively. The overall frequency of
tors examine 110 patients for the presence of a par- agreement p0 is given by p0 = 1/n (a11 + a22 + + akk), i.e., p0 equals the sum of
ticular disease and then state whether, in their the diagonal entries in the contingency table divided by the size of the sample.
opinion, each patient is or is not suffering from the Now, two stochastically independent raters would agree at a frequency pe that we
disease (the healthy/ill dichotomy). We wish to calculate as pe = a1./n a.1/n + a2./n a.2/n ++ a./n a.k/n , i.e., pe equals the
know how closely the two doctors diagnoses agree, sum of the products of the marginal frequencies relating to each diagonal entry of
i.e., how concordant they are. The 110 diagnoses of the general contingency table. Cohens kappa for k categories is defined just as it
the two doctors are shown in the Table in Box 1. was previously for k = 2, namely by the formula k = (p0-pe)/(1-pe) .
The two doctors arrived at the same diagnosis in
70 of 110 cases. This figure alone, however, is not
very useful in assessing concordance, because a cer- 1 2 ... k
tain number of like judgments would be expected 1 a11 a12 ... a1k a1.
even if one of the doctors (or both!) were assigning 2 a21 a22 ... a2k a2.
diagnoses entirely at random. On average, in this par-
... ... ... ... ... ...
ticular example, approximately 57 agreements would
be expected by chance alone, as explained in Box 1. k ak1 ak2 ... akk ak.
Cohens kappa reflects the difference between this a.1 a.2 ... a.k n
number (57) and the observed number of agreements
(70), in relation to the total number of cases (110). In
our example, Cohens kappa is 0.23. The value of
Cohens kappa would be 1 if the two raters agreed in
every case, i.e., if they were fully concordant; on the
other hand, a value of 0 would indicate that the two Box 2 generalizes the foregoing discussion to nom-
raters agreed only as often as they would by chance. inal rating scales with any number of categories, i.e.,
This would be a very discouraging finding indeed. A to scales with k 2. Until now, we have only been dis-
negative value of Cohens kappa would indicate that cussing dichotomous scales, for which, by definition,
the two raters agreed even less often than they would k = 2.
by chance, i.e., that they tended to make opposite Cohens kappa gives a quantitative assessment of
judgments. A value of 1 would mean that the two how well two raters agree, but the value of Cohens
raters arrived at opposite judgments in absolutely kappa itself conveys no information about the reliabil-
every case; this situation clearly arises very rarely, if ity of this assessment. A value determined from a small
ever. number of patients is unreliable; therefore, as in many
The interpretation of a statistic such as Cohens other situations in biomedical statistics, the value of
kappa is, in the end, arbitrary. Altman (2) suggested Cohens kappa should be stated together with the
categorizing values of Cohens kappa as shown in associated confidence interval (Box 3) (6).
Table 1. In the example of Box 1, the calculated value In practice, Cohens kappa is often used as a
2 = 0.23 would be considered to indicate no more one-sided test of whether the interrater agreement is
than a fair degree of agreement. strong enough to rule out random judgments by (at

Deutsches rzteblatt International | Dtsch Arztebl Int 2011; 108(30): 51521 519
MEDICINE

BOX 3 least) one rater. A statistically significant test result is

often wrongly interpreted as an objective indication of
agreement. In fact, this finding says next to nothing
The confidence interval for Cohens kappa about the degree of agreement between the two raters:
In general, the value of any descriptive statistic (e.g, a sample mean) conveys no even a very low positive value of Cohens kappa can be
information about its applicability to the overall population (not just to the sample statistically significant, as long as the sample on which
on which it is based). For this reason, descriptive statistics are usually reported it is based is large enough. Thus, the use of significance
with a confidence interval. An approximate 1- confidence interval for Cohens tests to judge concordance is a mistake.
kappa of the overall population can be calculated with the following formula: Cohens kappa can be further refined and general-
ized. When comparing ordinal ratings, one may wish to
give different weights to the differences between two
consecutive ratings (e.g., so that the difference between
good and excellent counts more than that between
In this formula, z1-/2 is the(1-/2) quantile of the standard normal distribution, fair and poor). A weighted kappa is used for this
whose values are listed in statistical tables (9). In the present case, we are ma- purpose. Interrater agreement can be assessed in other
king use of an approximation to the normal distribution. The following rule of situations, too, e.g., with more than two raters (7).
thumb is often helpful: such an approximation is acceptably accurate as long as Sensitivity and specificity are often used to compare
n p 0 5 and n (1-p0) 5. a dichotomous rating technique with a gold standard
We can now calculate a confidence interval for the numerical example presen- (8). These two statistics describe the degree of agree-
ted above in Box 1 and discussed in the corresponding section of the text. The ment between the technique in question and the gold
sample size is n = 110; furthermore, as we have already calculated, pe = 0.52, standard, in each of the two subpopulations that the
p0 = 0.63, and 2 = 0.23. Setting = 5%, we now read from a statistical table that gold standard defines. In contrast, Cohens kappa is a
z0.975 = 1.96. Using the formula above, we find: single quantity that provides a global assessment of the
agreement between the technique in question and the
CI = 0.23 1.96 0.0959 gold standard.

Equivalently stated, the 95% confidence interval for Cohens kappa of the overall Discussion
population is 0.042 to 0.418. Statistical methods of assessing the degree of agree-
ment between two raters or two measuring techniques
are used in two different situations:
ratings on a continuous scale, and
categorical (nominal) ratings.
In the first situation, it is advisable to use descriptive
and graphical methods, such as point-cloud plots
around the line of agreement and Bland-Altman
diagrams. Although point clouds are more intuitive and
perspicuous, Bland-Altman diagrams enable a more
detailed analysis in which the differences between the
two raters are assessed not just qualitatively, but also
quantitatively. The limits of agreement in a Bland-
Altman diagram may be unsuitable for assessing the
agreement between two measuring techniques if the
differences between measured values are not normally
KEY MESSAGES
distributed. In such cases, empirical quantiles can be
used instead.
The mere demonstration that a correlation coefficient The distribution of the differences between two
differs significantly from 0 is totally unsuitable for con- measured values can be studied in greater detail if, as
cordance analysis. Such tests are often wrongly used. first step, these differences are plotted on a histogram
(3). In many cases, when the two measuring techniques
The appropriate method for concordance analysis de- are linked by a good linear (or other functional) rela-
pends on the type of scale used by the measuring or
tionship, it will be possible to predict one of the
rating techniques that are to be compared.
measurements from the other one, even if the two tech-
The point-cloud diagram, the Bland-Altman diagram, and niques yield very different results at first glance. The
Cohens kappa are suitable methods for concordance Pearson correlation coefficient is a further type of
analysis. descriptive statistic; it indicates the presence of a linear
Concordance analysis cannot be used to judge the relationship. A significantly nonzero correlation coeffi-
correctness of measuring or rating techniques; rather, it cient, however, cannot be interpreted as implying that
shows the degree to which different measuring or rating two raters are concordant, as their ratings may still
techniques agree with each other. deviate from each other very strongly even when a
significant correlation is present.

520 Deutsches rzteblatt International | Dtsch Arztebl Int 2011; 108(30): 51521
 MEDICINE

Cohens kappa is a suitable tool for assessing the de- 4. Bland JM, Altman DG: Statistical methods for assessing agreement
gree of agreement between two raters for categorical between two methods of clinical measurement. Lancet 1986; 1:
30710.
(nominal) ratings. A confidence interval for Cohens
5. Song JW, Oh YM, Shim TS, Kim WS, Ryu JS, Choi CM: Efficacy
kappa can be calculated as well. comparison between (18)F-FDG PET/CT and bone scintigraphy in
detecting bony metastases of non-small-cell lung cancer. Lung
Conflict of interest statement
Cancer 2009; 65: 3338.
The authors declare that no conflict of interest exists. 6. du Prel JB, Hommel G, Rhrig B, Blettner M: Confidence interval or
p-value? Part 4 of a series on evaluation of scientific publications.
Manuscript submitted on 22 November 2010; revised version accepted on Dtsch Arztebl Int 2009; 106(19): 3359.
11 May 2011. 7. Bortz J, Lienert G A, Boehnke K: Verteilungsfreie Methoden in der
Biostatistik. 3rd Edition. Heidelberg: Springer 2008; 1929.
Translated from the original German by Ethan Taub, M.D.
8. Hilgers R D, Bauer P, Scheiber V: Einfhrung in die Medizinische
Statistik. 2nd edition. Heidelberg: Springer 2007.
REFERENCES 9. Altman DG, Machin D, Bryant TN, Gardner MJ: Statistics with confi-
dence. 2nd edition. London: BMJ Books 2000.
1. Tetzlaff R, Schwarz T, Kauczor HU, Meinzer HP, Puderbach M, Ei-
chinger M: Lung function measurement of single lungs by lung area
segmentation on 2D dynamic MRI. Acad Radiol. 2010; 17: Corresponding author
496503. Dr. rer. nat. Robert Kwiecien
2. Altman DG: Practical statistics for medical research. 1st edition. Institut fr Biometrie und Klinische Forschung (IBKF)
Westfhlische Wilhelms-Universitt Mnster
Oxford: Chapman and Hall 1991; 1611.
Albert-Schweitzer-Campus 1 Gebude A11
3. Altman DG, Bland JM: Measurement in medicine: the analysis of D-48149 Mnster, Germany
method comparison studies. The Statistician 1983; 32: 30717. [email protected]

Deutsches rzteblatt International | Dtsch Arztebl Int 2011; 108(30): 51521 521