How Response Designs and Class Proportions Affect the Accuracy of Validation Data

Assessing the relationship between morphology and mapping accuracy of built-up areas derived from global human settlement data

It is common knowledge that the level of landscape heterogeneity may affect the performance of remote sensing based land use / land cover classification. While this issue has been studied in depth for land cover data in general, the specific relationship between the mapping accuracy and morphological characteristics of built-up surfaces has not been analyzed in detail, an urgent need given the recent emergence of a variety of global, fine-resolution settlement datasets. Moreover, previous studies typically rely on aggregated, broad-scale landscape metrics to quantify the morphology of built-up areas, neglecting the fine-grained spatial variation and scale dependency of such metrics. Herein, we aim to fill this knowledge gap by assessing the associations between localized (focal) landscape metrics, derived from binary built-up surfaces and localized data accuracy estimates. We tested our approach for built-up surfaces from the Global Human Settlement Layer (GHSL) for Massachusetts (USA). Specifically, we examined the explanatory power of landscape metrics with respect to both commission and omission errors in the multi-temporal GHS-BUILT R2018A data product. We found that the Landscape Shape Index (LSI) calculated in focal windows exhibits, on average, the highest levels of correlation to focal accuracy measures. These relationships are scale-dependent, and become stronger with increasing level of spatial support. We found that thematic omission error, as measured by Recall, has the strongest relationship to measures of built-up surface morphology across different temporal epochs and spatial resolutions. The results of our regression analysis (R2>0.9), estimating accuracy based on landscape metrics, confirmed these findings. Lastly, we tested the generalizability of our findings by regionally stratifying our regression models and applying them to a different version of the GHSL (i.e., the GHS-BUILT-S2) and a different study area. We observed varying levels of model transferability, indicating that the relationship between accuracy and landscape metrics may be sensor-specific, and is heavily localized for most accuracy metrics, but quite generalizable for the Recall measure. This indicates that there is a strong and generalizable association between morphological properties of built-up land and the degree to which it is "undermapped".

About the Pitfall of Erroneous Validation Data in the Estimation of Confusion Matrices

Accuracy assessment of maps relies on the collection of validation data, i.e., a set of trusted points or spatial objects collected independently from the classified map. However, collecting spatially and thematically accurate dataset is often tedious and expensive. Despite good practices, those datasets are rarely error-prone. Errors in the reference dataset propagate to the probabilities estimated in the confusion matrices. Consequently, the estimates of the quality are biased: accuracy indices are overestimated if the errors are correlated and underestimated if the errors are conditionally independent. The first findings of our study highlight the fact that this bias could invalidate statistical tests of map accuracy assessment. Furthermore, correlated errors in the reference dataset induce unfair comparison of classifiers. A maximum entropy method is thus proposed to mitigate the propagation of errors from imperfect reference datasets. The proposed method is based on a theoretical framework which considers a trivariate probability table that links the observed confusion matrix, the confusion matrix of the reference dataset and the “real” confusion matrix. The method was tested with simulated thematic and geo-reference errors. It proved to reduce the bias to the level of the sampling uncertainty. The method was very efficient with geolocation errors because conditional independence of errors can reasonably be assumed. Thematic errors are more difficult to mitigate because they require the estimation of an additional parameter related to the amount of spatial correlation. In any case, while collecting additional trusted labels is usually expensive, our result show that the benefits for accuracy assessment are much larger than collecting a larger number of questionable reference data.

First 1-M Resolution Land Cover Map Labeling the Overlap in the 3rd Dimension: The 2018 Map for Wallonia

Land cover maps contribute to a large diversity of geospatial applications, including but not limited to land management, hydrology, land use planning, climate modeling and biodiversity monitoring. In densely populated and highly fragmented landscapes as observed in the Walloon region (Belgium), very high spatial resolution is required to depict all the infrastructures, buildings and most of the structural elements of the semi-natural landscapes (like hedges and small water bodies). Because of the resolution, the vertical dimension needs explicit handling to avoid discontinuities incompatible with many applications. For example, how to map a river flowing under a bridge? The particularity of our data is to provide a two-digit land cover code to label all the overlapping items. The identification of all the overlaps resulted from the combination of remote sensing image analysis and decision rules involving ancillary data. The final product is therefore semantically precise and accurate in terms of land cover description thanks to the addition of 24 classes on top of the 11 pure land cover classes. The quality of the map has been assessed using a state-of-the-art validation scheme. Its overall accuracy is as high as 91.5%, with an average producer’s accuracy of 86% and an average user’s accuracy of 91%.