EVT571 NEW (Lec 9 2) Accuracy Assessment

ENVIRONMENTAL REMOTE SENSING
(EVT571)
Lecture 9-2:
Classification Accuracy
Assessment
Dr. Arnis Asmat
Email: [email protected]
[email protected]
Introduction
Land-use/land-cover maps derived from remotely sensed
data are of little use if classification accuracy is not assessed.
Many studies ignore accuracy, or report an overall number

(e.g. 75%), which ignores locational accuracy.
Accuracy assessment involves comparison between:
A map derived from remotely sensed data; and
Reference test information.
The comparison is often made through an error matrix.

It is imperative that the error matrix be filled with unbiased
information.
The relationship between these two sets of information (a)

and (b) is commonly summarized in error matrix.
An error matrix is a square array of numbers laid out in rows

and column that expresses the number of sample units (i.e.,
pixels, clusters of pixels or polygon) assigned to a particular
category as verified in the field.
The columns normally represent the reference data while rows
indicate the classification generated from remotely sensed
data.
4
How do we tell if classification is any good?
Classification error matrix (aka confusion matrix or contingency table)

Need truth data sample pixels of known classes
How many pixels of KNOWN class X are incorrectly classified as anything other
than X (errors of omission)?
So-called Type 2 error, or false negative
Divide correctly classified pixels in each class of truth data by COLUMN
totals (Producers Accuracy)
How many pixels are incorrectly classified as class X when they should be some
other known class (errors of commission)?
So-called Type 1 error, or false positive
Divide correctly classified pixels in each class by ROW totals (Users
Accuracy)
5
Errors of
comission for
class U
Errors of
omission for
class U
Can use original training data to test BUT.

this only tells us how well the classifier can
classify the training areas
Ideally, use an independent set

of samples to give a better
'overall' accuracy estimate
6
Accuracy
Accuracy assessment
assessment
Comparison of a classification with ground-truth data to evaluate
how well the classification represents the real world.
Overall map The total number of correctly classified pixels that were
accuracy
sampled from image divided the total number of image
pixels sampled from the accuracy assessment procedure.
Users
accuracy
Probability that the land cover class an image pixel was

assigned to actually represent that category on the
ground.
Producers
accuracy
The probability that a reference pixel has been correctly

classified.
Individual
class
accuracy
Accuracy levels reported for an example an individual

land cover class (for example, the number of pixels that
was correctly classified when compared with the
reference data)
Example of classification accuracy report
Cont
Convert pixel to ha
Cont
Convert percent pixels to number of pixel
Training vs. Test Reference

Do not use training data to assess accuracy!
Instead, reference test pixels should be located.

These are sites for which land cover class is known but not
used in training.
They are sometimes collected before the classification, but

more commonly after.
The constraint of time is an important factor.
Sampling Size and Scheme

It is important to gather reference test pixels in sufficient
numbers to assess accuracy correctly.
Minimum of 50 samples for
each land cover class in the
error matrix.
Rule of thumb:
If the study area is large (>1

million acres) or there are a
large number of classes
(>12), increase the number
to 75-100 per class.
Cont
It is often appropriate to:
reduce the number of samples
needed in some classes (e.g.
water) and
increase the number in other,
more variable classes.
The goal is to balance the statistical need to
get a representative sample with:
time
cost
practical limitations
associated with the project
Cont
1. Most accuracy assessment statistics assume random
sampling.
2. But random sampling can under-sample small but

important classes.
3. Thus, stratified random sampling is usually employed.

4. This technique samples a minimum number of reference
pixels per land cover class.
Cont
5. After classification, the computer selects sites (geographic
x,y location) within each land cover class at random.
6. A site visit is then made (using GPS) to each to determine

reference land cover class.
7. Of course, this is not always possible aerial photos and

maps are often used as reference.
Analysis of spectral signatures

The uniqueness of spectral signatures necessary for deriving a
meaningful classification.
Spectral signatures
should:
be representative
of the entire
spectral space
have minimal
overlap with other
signatures
Uniqueness measured as separability
Spectral Separability
Spectral separability can be analysed using JeffriesMatusita Distance (J-M Distance)
Measured values were between 0 and 2, where:
0 indicated a complete
overlap between the
signatures of two classes
2 indicated a
complete separation
SIGNATURE SEPARABILITY REPORT FOR EIGHT LAND USE/COVER CLASSES

Table A: Signature separability summary
___________________________________________________________________________
Separability Measure
Jeffries-Matusita Distance
Signature Segments
38, 39, 40, 41, 42, 43, 44, and 45
Signature Channels
1, 2, 3, 4, 5, and 7
Average Separability
1.93
Minimum Separability
1.41
Maximum Separability
2.00
Signature Pair with Minimum Separability: (Mixed crops, Grasslands)
___________________________________________________________________________
Table B. Separability matrix for land use/cover classes
Category
Water Forest
Oil palm
Mixed
crops
Forest
2.00
Oil palm
2.00
1.68
Mixed crops
2.00
1.98
1.58
Grasslands
2.00
2.00
1.93
1.41
Rubber
2.00
1.88
1.89
1.85
Cleared
2.00
2.00
2.00
2.00
areas
Urban
1.95
2.00
2.00
2.00
Grasslands
Rubber
1.88
2.00
2.00
2.00
2.00
Cleared
areas
1.93
a. 1989
b. 1993
c. 1999
a. 1989
a. 1989
4 Kilometres
b. 1993
c. 1999
Water bodies
Forest
Oil palm
Mixed crops
Grasslands
Rubber
Cleared areas
Urban
b. 1993
4 Kilometres
Evaluation of Error Matrices

Once test reference data have been properly sampled, they
are compared pixel by pixel to the classification map.
Agreement or disagreement is summarized in the error
matrix.
Error matrix information is then evaluated using:
simple descriptive
statistics
and/or
multivariate
analytical statistics.

Overall accuracy =
total correct
total number of pixels in the error matrix
Example of Overall Accuracy
From Jensen (1996)

Producers accuracy =
total correct in a given class

total number of pixels in that class from reference data
Indicates the probability of a reference pixel being
classified correctly.
Called producers error because it tells the
producer of the map how well a certain class can be
classified.
Example of Producers Accuracy
Error of omission = error of exclusion
From Jensen (1996)

Users accuracy =
total correct in a given class
total number of pixels included in that class
Indicates the probability that a pixel classified on
the map actually represents that category on the
ground.
Called users error because it tells the user of the
map how accurate the classification is.
Example of Users Accuracy
Error of commission = error of inclusion

From Jensen (1996)

KAPPA analysis is a multivariate technique for accuracy
assessment.
It yields a Khat statistic that is a measure of agreement or

accuracy:
r
r
N xii xi x i
i 1
Khat i 1
r
2
N ( xi x i )
i 1
where r = the number of rows in the error matrix; xii = the number of
observations in row i and column i, xi+ and x+i are marginal totals for row
i and column i, and N is the total number of observations.
1.
Khat is used to determine if the results

presented in the error matrix are
significantly better than random class
assignment.
2.
It is also used to compare similar

classifications to determine if they are
significantly different.
3.
When reporting accuracy, include both

overall accuracy and Kappa!

EVT571 NEW (Lec 9 2) Accuracy Assessment

Uploaded by

Copyright:

Available Formats

EVT571 NEW (Lec 9 2) Accuracy Assessment

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

EVT571 NEW (Lec 9 2) Accuracy Assessment

Uploaded by

Copyright:

Available Formats

ENVIRONMENTAL REMOTE SENSING

Many studies ignore accuracy, or report an overall number

The comparison is often made through an error matrix.

The relationship between these two sets of information (a)

An error matrix is a square array of numbers laid out in rows

How do we tell if classification is any good?

Classification error matrix (aka confusion matrix or contingency table)

Can use original training data to test BUT.

Ideally, use an independent set

Probability that the land cover class an image pixel was

The probability that a reference pixel has been correctly

Accuracy levels reported for an example an individual

Example of classification accuracy report

Convert percent pixels to number of pixel

Training vs. Test Reference

Instead, reference test pixels should be located.

They are sometimes collected before the classification, but

Sampling Size and Scheme

If the study area is large (>1

2. But random sampling can under-sample small but

3. Thus, stratified random sampling is usually employed.

6. A site visit is then made (using GPS) to each to determine

7. Of course, this is not always possible aerial photos and

Analysis of spectral signatures

Uniqueness measured as separability

SIGNATURE SEPARABILITY REPORT FOR EIGHT LAND USE/COVER CLASSES

Evaluation of Error Matrices

Evaluation of Error Matrices

Example of Overall Accuracy

From Jensen (1996)

Evaluation of Error Matrices

total correct in a given class

Example of Producers Accuracy

Error of omission = error of exclusion

From Jensen (1996)

Evaluation of Error Matrices

Example of Users Accuracy

Error of commission = error of inclusion

Evaluation of Error Matrices

It yields a Khat statistic that is a measure of agreement or

Evaluation of Error Matrices

Khat is used to determine if the results

It is also used to compare similar

When reporting accuracy, include both

You might also like