Quantifying Error in Training Data for Mapping and Monitoring the Earth System - A Workshop on “Quantifying Error in Training Data for Mapping and Monitoring the Earth System” was held on January 8-9, 2019 at Clark University, with support from Omidyar Network’s Property Rights Initiative, now PlaceFund.

1. Assessing land cover change
using uncertain data
Robert Gilmore Pontius Jr
rpontius@clarku.edu
www.clarku.edu/~rpontius
8-9 January 2019
Workshop on
Quantifying Error in Training Data and its Implications for Land Cover Mapping
1

2. Agreement between classification and truth
must be between these bounds
0
25
50
75
100
0
25
50
75
100
0
25
50
75
100
agreementoftruthwithmodel
agreement of truth with data
agreement of
model with data
75-100
50-75
25-50
0-25
0
25
50
75
100
0
25
50
75
100
0
25
50
75
100
agreementoftruthwithmodel
agreement of truth with data
agreement of
model with data
75-100
50-75
25-50
0-25
2
Pontius Jr, Robert Gilmore, and Silvia H. Petrova. 2010. “Assessing a Predictive Model of Land Change
Using Uncertain Data.” Environmental Modelling & Software 25 (3): 299–309.
https://doi.org/10.1016/j.envsoft.2009.09.005.
agreement of truth with data
agreement of classification
with data
agreementofclassification
withtruth
agreement of truth with data
agreement of classification
with data
agreementofclassification
withtruth
Lower Bound Upper Bound

3. Two groups that caught Gil’s attention
• Group 1: State of knowledge on error and its quantification:
• Group focus: Summarizing the current state of knowledge regarding
quantification of: i) error in land cover classification and other relevant
remote measurement methods (e.g. maps with non-discrete values); ii) error
in training data and its propagation into maps.
• Group 1: A complete accuracy metric
• Group focus: Prototype a metric that accounts for training data error in
assessment of map accuracy.
3

4. Difference Components for a categorical variable:
Quantity, Exchange and Shift
4
Reference
j=1 j=2 j=3 j=4 Sum
Comparison
i=1 0 0 2 0 2
i=2 0 0 0 3 3
i=3 2 3 0 0 5
i=4 0 0 0 0 0
Sum 2 3 2 3 10
1 1
2 2
3 3
3 3
2 3
Comparison
3 3
4 4
1 1
2 2
4 2
Reference
Dashed lines indicate overall difference.

5. Pop Quiz 1 for continuous variable
What is the difference between 6 and 5?
5

6. Pop Quiz 2 for continuous variable
What is the agreement between 6 and 5?
6

7. 7
Satellite measurement versus field measurement
R2 measures whether points form any line.
R2 does not indicate how far points are from Y=X line.

8. 8
Satellite measurement versus field measurement
Mean Deviation measures how far the average is above or below the Y=X line.
Mean Absolute Deviation measures how far points are from Y=X line.
See videos at https://www2.clarku.edu/faculty/rpontius/videos.html
0.044
0.0331.224
-1.108

9. Literature
1. Olofsson, Pontus, Giles M. Foody, Martin Herold, Stephen V. Stehman, Curtis E. Woodcock, and Michael A. Wulder. 2014. “Good Practices for
Estimating Area and Assessing Accuracy of Land Change.” Remote Sensing of Environment 148 (May): 42–57.
https://doi.org/10.1016/j.rse.2014.02.015.
2. Oort, P.A.J. van. 2007. “Interpreting the Change Detection Error Matrix.” Remote Sensing of Environment 108 (1): 1–8.
https://doi.org/10.1016/j.rse.2006.10.012.
3. Pontius Jr, Robert Gilmore, and Marco Millones. 2011. “Death to Kappa: Birth of Quantity Disagreement and Allocation Disagreement for
Accuracy Assessment.” International Journal of Remote Sensing 32 (15): 4407–29. https://doi.org/10.1080/01431161.2011.552923.
4. Pontius Jr, Robert Gilmore, and Silvia H. Petrova. 2010. “Assessing a Predictive Model of Land Change Using Uncertain Data.” Environmental
Modelling & Software 25 (3): 299–309. https://doi.org/10.1016/j.envsoft.2009.09.005.
5. Pontius Jr, Robert Gilmore, and Alí Santacruz. 2014. “Quantity, Exchange, and Shift Components of Difference in a Square Contingency Table.”
International Journal of Remote Sensing 35 (21): 7543–54. https://doi.org/10.1080/2150704X.2014.969814.
6. ———. 2015. DiffeR: Metrics of Difference for Comparing Pairs of Maps (version 0.0-4). R. amsantac.org. https://cran.r-
project.org/web/packages/diffeR.
7. Pontius Jr, Robert Gilmore, Olufunmilayo Thontteh, and Hao Chen. 2008. “Components of Information for Multiple Resolution Comparison
between Maps That Share a Real Variable.” Environmental and Ecological Statistics 15 (2): 111–42. https://doi.org/10.1007/s10651-007-0043-y.
8. Willmott, C. J., and K. Matsuura. 2006. “On the Use of Dimensioned Measures of Error to Evaluate the Performance of Spatial Interpolators.”
International Journal of Geographical Information Science 20 (1): 89–102. https://doi.org/10.1080/13658810500286976.
9. Ye, Su, Robert Gilmore Pontius Jr, and Rahul Rakshit. 2018. “A Review of Accuracy Assessment for Object-Based Image Analysis: From per-Pixel to
per-Polygon Approaches.” ISPRS Journal of Photogrammetry and Remote Sensing 141: 137–47.
9

10. Major Points to avoid common blunders
1. Focus on components of difference, rather than one metric of
agreement.
• For a categorical variable, compute components of Quantity, Exchange and Shift.
• For a continuous variable, compute Mean Deviation and Mean Absolute Deviation.
2. Let the research questions dictate the selection of the metrics.
3. Compare results to a baseline of an alternative mapping method,
rather than to a socially-constructed criterion for acceptability.
4. Focus on error of change during a time interval, rather than the
accuracy at specific time points.
10

11. X Y
11

12. 12

13. 13
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
1.6
1.8
2.0
2.2
2.4
2.6
2.8
3.0
3.2
3.4
3.6
3.8
4.0
0.1 0.5 1.5 4.5
MeanAbsoluteDeviation(indegreesCelcius)
Spatial Resolution(in thousand kilometers per pixel's side)
Mean of August 1982 < Mean of February 1982
Pixel-level Allocation
Stratum-level Allocation
Quantity

14. Deviations at four spatial resolutions
14

15. 15
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
1.6
1.8
2.0
2.2
2.4
2.6
2.8
3.0
3.2
3.4
3.6
3.8
4.0
0.1 0.5 1.5 4.5
MeanAbsoluteDeviation(indegreesCelcius)
Spatial Resolution(in thousand kilometers per pixel's side)
Mean of August 1982 < Mean of February 1982
Pixel-level Allocation
Stratum-level Allocation
Quantity

16. Critique this figure (Ewa et al. 2012)
16

17. Critique this figure (Ewa et al. 2012)
17