Multispectral high resolution sensor fusion for smoothing and gap-filling in the cloud

A 30-m annual corn residue coverage dataset from 2013 to 2021 in Northeast China

Crop residue cover plays a key role in the protection of black soil by covering the soil in the non-growing season against wind erosion and chopping for returning to the soil to increase organic matter in the future. Although there are some studies that have mapped the crop residue coverage by remote sensing technique, the results are mainly on a small scale, limiting the generalizability of the results. In this study, we present a novel corn residue coverage (CRC) dataset for Northeast China spanning the years 2013-2021. The aim of our dataset is to provide a basis to describe and monitor CRC for black soil protection. The accuracy of our estimation results was validated against previous studies and measured data, demonstrating high accuracy with a coefficient of determination (R2) of 0.7304 and root mean square error (RMSE) of 0.1247 between estimated and measured CRC in field campaigns. In addition, it is the first of its kind to offer the longest time series, enhancing its significance in long-term monitoring and analysis.

Combination of Sentinel-1 and Sentinel-2 Data for Tree Species Classification in a Central European Biosphere Reserve

Microwave and optical imaging methods react differently to different land surface parameters and, thus, provide highly complementary information. However, the contribution of individual features from these two domains of the electromagnetic spectrum for tree species classification is still unclear. For large-scale forest assessments, it is moreover important to better understand the domain-specific limitations of the two sensor families, such as the impact of cloudiness and low signal-to-noise-ratio, respectively. In this study, seven deciduous and five coniferous tree species of the Austrian Biosphere Reserve Wienerwald (105,000 ha) were classified using Breiman’s random forest classifier, labeled with help of forest enterprise data. In nine test cases, variations of Sentinel-1 and Sentinel-2 imagery were passed to the classifier to evaluate their respective contributions. By solely using a high number of Sentinel-2 scenes well spread over the growing season, an overall accuracy of 83.2% was achieved. With ample Sentinel-2 scenes available, the additional use of Sentinel-1 data improved the results by 0.5 percentage points. This changed when only a single Sentinel-2 scene was supposedly available. In this case, the full set of Sentinel-1-derived features increased the overall accuracy on average by 4.7 percentage points. The same level of accuracy could be obtained using three Sentinel-2 scenes spread over the vegetation period. On the other hand, the sole use of Sentinel-1 including phenological indicators and additional features derived from the time series did not yield satisfactory overall classification accuracies (55.7%), as only coniferous species were well separated.

Multi-Season Phenology Mapping of Nile Delta Croplands Using Time Series of Sentinel-2 and Landsat 8 Green LAI

Space-based cropland phenology monitoring substantially assists agricultural managing practices and plays an important role in crop yield predictions. Multitemporal satellite observations allow analyzing vegetation seasonal dynamics over large areas by using vegetation indices or by deriving biophysical variables. The Nile Delta represents about half of all agricultural lands of Egypt. In this region, intensifying farming systems are predominant and multi-cropping rotations schemes are increasing, requiring a high temporal and spatial resolution monitoring for capturing successive crop growth cycles. This study presents a workflow for cropland phenology characterization and mapping based on time series of green Leaf Area Index (LAI) generated from NASA’s Harmonized Landsat 8 (L8) and Sentinel-2 (S2) surface reflectance dataset from 2016 to 2019. LAI time series were processed for each satellite dataset, which were used separately and combined to identify seasonal dynamics for a selection of crop types (wheat, clover, maize and rice). For the combination of L8 with S2 LAI products, we proposed two time series smoothing and fitting methods: (1) the Savitzky–Golay (SG) filter and (2) the Gaussian Processes Regression (GPR) fitting function. Single-sensor and L8-S2 combined LAI time series were used for the calculation of key crop Land Surface Phenology (LSP) metrics (start of season, end of season, length of season), whereby the detection of cropland growing seasons was based on two established threshold methods, i.e., a seasonal or a relative amplitude value. Overall, the developed phenology extraction scheme enabled identifying up to two successive crop cycles within a year, with a superior performance observed for the seasonal than for the relative threshold method, in terms of consistency and cropland season detection capability. Differences between the time series collections were analyzed by comparing the phenology metrics per crop type and year. Results suggest that L8-S2 combined LAI data streams with GPR led to a more precise detection of the start and end of growing seasons for most crop types, reaching an overall detection of 74% over the total planted crops versus 69% with S2 and 63% with L8 alone. Finally, the phenology mapping allowed us to evaluate the spatial and temporal evolution of the croplands over the agroecosystem in the Nile Delta.

An NDVI Retrieval Method Based on a Double-Attention Recurrent Neural Network for Cloudy Regions

NDVI is an important parameter for environmental assessment and precision agriculture that well-describes the status of vegetation. Nevertheless, the clouds in optical images often result in the absence of NDVI information at key growth stages. The integration of SAR and optical image features will likely address this issue. Although the mapping of different data sources is complex, the prosperity of deep learning technology provides an alternative approach. In this study, the double-attention RNN architecture based on the recurrent neural network (RNN) and attention mechanism is proposed to retrieve NDVI data of cloudy regions. Overall, the NDVI is retrieved by the proposed model from two aspects: the temporal domain and the pixel neighbor domain. The performance of the double-attention RNN is validated through different cloud coverage conditions, input ablation, and comparative experiments with various methods. The results conclude that a high retrieval accuracy is guaranteed by the proposed model, even under high cloud coverage conditions (R2 = 0.856, RMSE = 0.124). Using SAR images independently results in poor NDVI retrieval results (R2 = 0.728, RMSE = 0.141) with considerable artifacts, which need to be addressed with auxiliary data, such as IDM features. Temporal and pixel neighbor features play an important role in improving the accuracy of NDVI retrieval (R2 = 0.894, RMSE = 0.096). For the missing values of NDVI data caused by cloud coverage, the double-attention RNN proposed in this study provides a potential solution for information recovery.

An object-based sparse representation model for spatiotemporal image fusion

Many algorithms have been proposed for spatiotemporal image fusion on simulated data, yet only a few deal with spectral changes in real satellite images. An innovative spatiotemporal sparse representation (STSR) image fusion approach is introduced in this study to generate global dense high spatial and temporal resolution images from real satellite images. It aimed to minimize the data gap, especially when fine spatial resolution images are unavailable for a specific period. The proposed approach uses a set of real coarse- and fine-spatial resolution satellite images acquired simultaneously and another coarse image acquired at a different time to predict the corresponding unknown fine image. During the fusion process, pixels located between object classes with different spectral responses are more vulnerable to spectral distortion. Therefore, firstly, a rule-based fuzzy classification algorithm is used in STSR to classify input data and extract accurate edge candidates. Then, an object-based estimation of physical constraints and brightness shift between input data is utilized to construct the proposed sparse representation (SR) model that can deal with real input satellite images. Initial rules to adjust spatial covariance and equalize spectral response of object classes between input images are introduced as prior information to the model, followed by an optimization step to improve the STSR approach. The proposed method is applied to real fine Sentinel-2 and coarse Landsat-8 satellite data. The results showed that introducing objects in the fusion process improved spatial detail, especially over the edge candidates, and eliminated spectral distortion by preserving the spectral continuity of extracted objects. Experiments revealed the promising performance of the proposed object-based STSR image fusion approach based on its quantitative results, where it preserved almost 96.9% and 93.8% of the spectral detail over the smooth and urban areas, respectively.

DroughtCast: A Machine Learning Forecast of the United States Drought Monitor

Drought is one of the most ecologically and economically devastating natural phenomena affecting the United States, causing the U.S. economy billions of dollars in damage, and driving widespread degradation of ecosystem health. Many drought indices are implemented to monitor the current extent and status of drought so stakeholders such as farmers and local governments can appropriately respond. Methods to forecast drought conditions weeks to months in advance are less common but would provide a more effective early warning system to enhance drought response, mitigation, and adaptation planning. To resolve this issue, we introduce DroughtCast, a machine learning framework for forecasting the United States Drought Monitor (USDM). DroughtCast operates on the knowledge that recent anomalies in hydrology and meteorology drive future changes in drought conditions. We use simulated meteorology and satellite observed soil moisture as inputs into a recurrent neural network to accurately forecast the USDM between 1 and 12 weeks into the future. Our analysis shows that precipitation, soil moisture, and temperature are the most important input variables when forecasting future drought conditions. Additionally, a case study of the 2017 Northern Plains Flash Drought shows that DroughtCast was able to forecast a very extreme drought event up to 12 weeks before its onset. Given the favorable forecasting skill of the model, DroughtCast may provide a promising tool for land managers and local governments in preparing for and mitigating the effects of drought.

Spectral Temporal Information for Missing Data Reconstruction (STIMDR) of Landsat Reflectance Time Series

The number of Landsat time-series applications has grown substantially because of its approximately 50-year history and relatively high spatial resolution for observing long term changes in the Earth’s surface. However, missing observations (i.e., gaps) caused by clouds and cloud shadows, orbit and sensing geometry, and sensor issues have broadly limited the development of Landsat time-series applications. Due to the large area and temporal and spatial irregularity of time-series gaps, it is difficult to find an efficient and highly precise method to fill them. The Missing Observation Prediction based on Spectral-Temporal Metrics (MOPSTM) method has been proposed and delivered good performance in filling large-area gaps of single-date Landsat images. However, it can be less practical for a time series longer than one year due to the lack of mechanics that exclude dissimilar data in time series (e.g., different phenology or changes in land cover). To solve this problem, this study proposes a new gap-filling method, Spectral Temporal Information for Missing Data Reconstruction (STIMDR), and examines its performance in Landsat reflectance time series. Two groups of experiments, including 2000 × 2000 pixel Landsat single-date images and Landsat time series acquired from four sites (Kenya, Finland, Germany, and China), were performed to test the new method. We simulated artificial gaps to evaluate predicted pixel values with real observations. Quantitative and qualitative evaluations of gap-filled images through comparisons with other state-of-the-art methods confirmed the more robust and accurate performance of the proposed method. In addition, the proposed method was also able to fill gaps contaminated by extreme cloud cover for a period (e.g., winter in high-latitude areas). A down-stream task of random forest supervised classification through both gap-filled simulated datasets and the original valid datasets verified that STIMDR-generated products are relevant to the user community for land cover applications.

ERTFM: An Effective Model to Fuse Chinese GF-1 and MODIS Reflectance Data for Terrestrial Latent Heat Flux Estimation

Coarse spatial resolution sensors play a major role in capturing temporal variation, as satellite images that capture fine spatial scales have a relatively long revisit cycle. The trade-off between the revisit cycle and spatial resolution hinders the access of terrestrial latent heat flux (LE) data with both fine spatial and temporal resolution. In this paper, we firstly investigated the capability of an Extremely Randomized Trees Fusion Model (ERTFM) to reconstruct high spatiotemporal resolution reflectance data from a fusion of the Chinese GaoFen-1 (GF-1) and the Moderate Resolution Imaging Spectroradiometer (MODIS) products. Then, based on the merged reflectance data, we used a Modified-Satellite Priestley–Taylor (MS–PT) algorithm to generate LE products at high spatial and temporal resolutions. Our results illustrated that the ERTFM-based reflectance estimates showed close similarity with observed GF-1 images and the predicted NDVI agreed well with observed NDVI at two corresponding dates (r = 0.76 and 0.86, respectively). In comparison with other four fusion methods, including the widely used spatial and temporal adaptive reflectance fusion model (STARFM) and the enhanced STARFM, ERTFM had the best performance in terms of predicting reflectance (SSIM = 0.91; r = 0.77). Further analysis revealed that LE estimates using ERTFM-based data presented more detailed spatiotemporal characteristics and provided close agreement with site-level LE observations, with an R2 of 0.81 and an RMSE of 19.18 W/m2. Our findings suggest that the ERTFM can be used to improve LE estimation with high frequency and high spatial resolution, meaning that it has great potential to support agricultural monitoring and irrigation management.

Assessment of the Spatial and Temporal Patterns of Cover Crops Using Remote Sensing

Cover cropping is a conservation practice that helps to alleviate soil health problems and reduce nutrient losses. Understanding the spatial variability in historic and current adoption of cover cropping practices and their impacts on soil, water, and nutrient dynamics at a landscape scale is an important step in determining and prioritizing areas in a watershed to effectively utilize this practice. However, such data are lacking. Our objective was to develop a spatial and temporal inventory of winter cover cropping practices in the Maumee River watershed using images collected by Landsat satellites (Landsat 5, 7 and 8) from 2008 to 2019 in Google Earth Engine (GEE) platform. Each year, satellite images collected during cover crop growing season (i.e., between October and April) were converted into two seasonal composites based on cover crop phenology. Using these composites, various image-based covariates were extracted for 628 ground-truth (field) data. By integrating ground-truth and image-based covariates, a cover crop classification model based on a random forest (RF) algorithm was developed, trained and validated in GEE platform. Our classification scheme differentiated four cover crop categories: Winter Hardy, Winter Kill, Spring Emergent, and No Cover. The overall classification accuracy was 75%, with a kappa coefficient of 0.63. The results showed that more than 50% of the corn-soybean areas in the Maumee River watershed were without winter crops during 2008–2019 period. It was found that 2019/2020 and 2009/2010 were the years with the largest and lowest cover crop areas, with 34% and 10% in the watershed, respectively. The total cover cropping area was then assessed in relation to fall precipitation and cumulative growing degree days (GDD). There was no apparent increasing trend in cover crop areas between 2008 and 2019, but the variability in cover crops areas was found to be related to higher accumulated GDD and fall precipitation. A detailed understanding of the spatial and temporal distribution of cover crops using GEE could help in promoting site-specific management practices to enhance their environmental benefits. This also has significance to policy makers and funding agencies as they could use the information to localize areas in need of interventions for supporting adoption of cover cropping practice.

Mask R-CNN and OBIA Fusion Improves the Segmentation of Scattered Vegetation in Very High-Resolution Optical Sensors

Vegetation generally appears scattered in drylands. Its structure, composition and spatial patterns are key controls of biotic interactions, water, and nutrient cycles. Applying segmentation methods to very high-resolution images for monitoring changes in vegetation cover can provide relevant information for dryland conservation ecology. For this reason, improving segmentation methods and understanding the effect of spatial resolution on segmentation results is key to improve dryland vegetation monitoring. We explored and analyzed the accuracy of Object-Based Image Analysis (OBIA) and Mask Region-based Convolutional Neural Networks (Mask R-CNN) and the fusion of both methods in the segmentation of scattered vegetation in a dryland ecosystem. As a case study, we mapped Ziziphus lotus, the dominant shrub of a habitat of conservation priority in one of the driest areas of Europe. Our results show for the first time that the fusion of the results from OBIA and Mask R-CNN increases the accuracy of the segmentation of scattered shrubs up to 25% compared to both methods separately. Hence, by fusing OBIA and Mask R-CNNs on very high-resolution images, the improved segmentation accuracy of vegetation mapping would lead to more precise and sensitive monitoring of changes in biodiversity and ecosystem services in drylands.

A Simple Spatio–Temporal Data Fusion Method Based on Linear Regression Coefficient Compensation

High spatio–temporal resolution remote sensing images are of great significance in the dynamic monitoring of the Earth’s surface. However, due to cloud contamination and the hardware limitations of sensors, it is difficult to obtain image sequences with both high spatial and temporal resolution. Combining coarse resolution images, such as the moderate resolution imaging spectroradiometer (MODIS), with fine spatial resolution images, such as Landsat or Sentinel-2, has become a popular means to solve this problem. In this paper, we propose a simple and efficient enhanced linear regression spatio–temporal fusion method (ELRFM), which uses fine spatial resolution images acquired at two reference dates to establish a linear regression model for each pixel and each band between the image reflectance and the acquisition date. The obtained regression coefficients are used to help allocate the residual error between the real coarse resolution image and the simulated coarse resolution image upscaled by the high spatial resolution result of the linear prediction. The developed method consists of four steps: (1) linear regression (LR), (2) residual calculation, (3) distribution of the residual and (4) singular value correction. The proposed method was tested in different areas and using different sensors. The results show that, compared to the spatial and temporal adaptive reflectance fusion model (STARFM) and the flexible spatio–temporal data fusion (FSDAF) method, the ELRFM performs better in capturing small feature changes at the fine image scale and has high prediction accuracy. For example, in the red band, the proposed method has the lowest root mean square error (RMSE) (ELRFM: 0.0123 vs. STARFM: 0.0217 vs. FSDAF: 0.0224 vs. LR: 0.0221). Furthermore, the lightweight algorithm design and calculations based on the Google Earth Engine make the proposed method computationally less expensive than the STARFM and FSDAF.

A Machine Learning Approach for Remote Sensing Data Gap-Filling with Open-Source Implementation: An Example Regarding Land Surface Temperature, Surface Albedo and NDVI

Satellite remote sensing has now become a unique tool for continuous and predictable monitoring of geosystems at various scales, observing the dynamics of different geophysical parameters of the environment. One of the essential problems with most satellite environmental monitoring methods is their sensitivity to atmospheric conditions, in particular cloud cover, which leads to the loss of a significant part of data, especially at high latitudes, potentially reducing the quality of observation time series until it is useless. In this paper, we present a toolbox for filling gaps in remote sensing time-series data based on machine learning algorithms and spatio-temporal statistics. The first implemented procedure allows us to fill gaps based on spatial relationships between pixels, obtained from historical time-series. Then, the second procedure is dedicated to filling the remaining gaps based on the temporal dynamics of each pixel value. The algorithm was tested and verified on Sentinel-3 SLSTR and Terra MODIS land surface temperature data and under different geographical and seasonal conditions. As a result of validation, it was found that in most cases the error did not exceed 1 °C. The algorithm was also verified for gaps restoration in Terra MODIS derived normalized difference vegetation index and land surface broadband albedo datasets. The software implementation is Python-based and distributed under conditions of GNU GPL 3 license via public repository.

A Framework of Spatio-Temporal Fusion Algorithm Selection for Landsat NDVI Time Series Construction

Spatio-temporal fusion algorithms dramatically enhance the application of the Landsat time series. However, each spatio-temporal fusion algorithm has its pros and cons of heterogeneous land cover performance, the minimal number of input image pairs, and its efficiency. This study aimed to answer: (1) how to determine the adaptability of the spatio-temporal fusion algorithm for predicting images in prediction date and (2) whether the Landsat normalized difference vegetation index (NDVI) time series would benefit from the interpolation with images fused from multiple spatio-temporal fusion algorithms. Thus, we supposed a linear relationship existed between the fusion accuracy and spatial and temporal variance. Taking the Spatial and Temporal Adaptive Reflectance Fusion Model (STARFM) and the Enhanced STARFM (ESTARFM) as basic algorithms, a framework was designed to screen a spatio-temporal fusion algorithm for the Landsat NDVI time series construction. The screening rule was designed by fitting the linear relationship between the spatial and temporal variance and fusion algorithm accuracy, and then the fitted relationship was combined with the graded accuracy selecting rule (R2) to select the fusion algorithm. The results indicated that the constructed Landsat NDVI time series by this paper proposed framework exhibited the highest overall accuracy (88.18%), and lowest omission (1.82%) and commission errors (10.00%) in land cover change detection compared with the moderate resolution imaging spectroradiometer (MODIS) NDVI time series and the NDVI time series constructed by a single STARFM or ESTARFM. Phenological stability analysis demonstrated that the Landsat NDVI time series established by multiple spatio-temporal algorithms could effectively avoid phenological fluctuations in the time series constructed by a single fusion algorithm. We believe that this framework can help improve the quality of the Landsat NDVI time series and fulfill the gap between near real-time environmental monitoring mandates and data-scarcity reality.