Spatiotemporal Fusion of Multisource Remote Sensing Data: Literature Survey, Taxonomy, Principles, Applications, and Future Directions

Integrated UAV and Satellite Multi-Spectral for Agricultural Drought Monitoring of Winter Wheat in the Seedling Stage

Agricultural droughts are a threat to local economies, as they disrupt crops. The monitoring of agricultural droughts is of practical significance for mitigating loss. Even though satellite data have been extensively used in agricultural studies, realizing wide-range, high-resolution, and high-precision agricultural drought monitoring is still difficult. This study combined the high spatial resolution of unmanned aerial vehicle (UAV) remote sensing with the wide-range monitoring capability of Landsat-8 and employed the local average method for upscaling to match the remote sensing images of the UAVs with satellite images. Based on the measured ground data, this study employed two machine learning algorithms, namely, random forest (RF) and eXtreme Gradient Boosting (XGBoost1.5.1), to establish the inversion models for the relative soil moisture. The results showed that the XGBoost model achieved a higher accuracy for different soil depths. For a soil depth of 0-20 cm, the XGBoost model achieved the optimal result (R2 = 0.6863; root mean square error (RMSE) = 3.882%). Compared with the corresponding model for soil depth before the upscaling correction, the UAV correction can significantly improve the inversion accuracy of the relative soil moisture according to satellite remote sensing. To conclude, a map of the agricultural drought grade of winter wheat in the Huaibei Plain in China was drawn up.

Comparison of change-based and shape-based data fusion methods in fine-resolution land surface phenology monitoring with Landsat and Sentinel-2 data

Fine-resolution land surface phenology (LSP) is urgently required for applications on agriculture management and vegetation-climate interaction, especially over heterogeneous areas, such as agricultural lands and fragmented forests. The critical challenge of fine-resolution LSP monitoring is how to reconstruct the spatiotemporal continuous vegetation index time series. To solve this problem, various data fusion methods have been devised; however, the comprehensive inter-comparison is lacking across different spatial heterogeneity, data quality, and vegetation types. We divide these methods into two main categories: the change-based methods fusing satellite observations with different spatiotemporal resolutions, and the shape-based methods fusing prior knowledge of shape models and satellite observations. We selected four methods to rebuilt two-band enhanced vegetation index (EVI2) series based on the harmonized Landsat and Sentinel-2 (HLS) data, including two change-based methods, namely the Spatial and temporal Adaptive Reflectance Fusion Model (STARFM), the Flexible Spatiotemporal DAta Fusion (FSDAF), and two shape-based methods, namely the Multiple-year Weighting Shape-Matching (MWSM), and the Spatiotemporal Shape-Matching Model (SSMM). Four phenological transition dates were extracted, evaluated with PhenoCam observations and the 500 m Visible Infrared Imaging Radiometer Suite (VIIRS) phenology product. The 30 m transition dates show more spatial details and reveal more apparent intra-class and inter-class phenology variation compared with 500 m product. The four transition dates of SSMM and FSDAF (R2>0.74, MAD<15 days) show better agreement with PhenoCam-derived dates. The performance difference between fusion methods over various application scenarios are then analyzed. Fusion results are more robust when temporal frequency is higher than 15 observations per year. The shape-based methods are less sensitive to temporal sampling irregularity than change-based methods. Both change-based methods and shape-based methods cannot perform well when the region is heterogeneous. Among different vegetation types, SSMM-like methods have the highest overall accuracy. The findings in this paper can provide references for regional and global fine-resolution phenology monitoring.

Impact of rainfed agriculture on spatio-temporal patterns of water balance and the interaction between groundwater and surface water in sub-humid plains

The expansion of rainfed agriculture, especially soybean cultivation in sub-humid plains, alters water balance and the exchange between groundwater-surface water (GW-SW). However, to date, there are no studies that analyze how these anthropic disturbances affect hydrological connectivity in these systems, especially the GW-SW interactions. The objective of this study is to analyze how the increase in rainfed agriculture affects the spatio-temporal patterns of the water balance and the GW-SW interaction. For this analysis, a coupled GW-SW flow model was implemented under land use and land cover (LULC) scenarios, to quantify the spatio-temporal dynamics for different components of water balance and GW-SW interactions for the upper creek basin of Del Azul. A simulation was carried out for a period of 13 years (2003-2015) on a daily scale and it was contrasted through three multitemporal LULC maps. The results point that substitution of natural pastures, the reduction of winter crops and the decrease of crop rotation, due to the increase of soybean monoculture in the basin under study, modifies the water balance, especially the annual rates of surface runoff and soil moisture which may increase between 3.5 and 9.4 % and between 1.4 and 4.4 % respectively, thus increasing the annual streamflows between 2.6 and 6.8 % and the groundwater heads between 0.2 and 0.6 m. This leads to changes in the hetereogeneity of the GW-SW interaction, a reduction between 0.3 and 3 % is observed in the discharge from the Pampeano aquifer to the Del Azul stream, while the recharge rates from the Del Azul stream to the Pampeano aquifer increase between 2 and 17.8 %. The application of the SWAT-MODFLOW model under LULC scenarios, improves the prediction of the regional hydrologic connectivity on sub-humid plains, because the hydrological processes occurring in the surface and non-saturated zone are governed by shallow groundwater dynamics.

Fractional Fourier Image Transformer for Multimodal Remote Sensing Data Classification

With the recent development of the joint classification of hyperspectral image (HSI) and light detection and ranging (LiDAR) data, deep learning methods have achieved promising performance owing to their locally sematic feature extracting ability. Nonetheless, the limited receptive field restricted the convolutional neural networks (CNNs) to represent global contextual and sequential attributes, while visual image transformers (VITs) lose local semantic information. Focusing on these issues, we propose a fractional Fourier image transformer (FrIT) as a backbone network to extract both global and local contexts effectively. In the proposed FrIT framework, HSI and LiDAR data are first fused at the pixel level, and both multisource feature and HSI feature extractors are utilized to capture local contexts. Then, a plug-and-play image transformer FrIT is explored for global contextual and sequential feature extraction. Unlike the attention-based representations in classic VIT, FrIT is capable of speeding up the transformer architectures massively and learning valuable contextual information effectively and efficiently. More significantly, to reduce redundancy and loss of information from shallow to deep layers, FrIT is devised to connect contextual features in multiple fractional domains. Five HSI and LiDAR scenes including one newly labeled benchmark are utilized for extensive experiments, showing improvement over both CNNs and VITs.

Refined assessment of space-time changes, influencing factors and socio-economic impacts of the terrestrial ecosystem quality: A case study of the GBA

The terrestrial ecosystem is the cradle of energy and material basis for human survival and development. However, there are large research deficits in accurately and finely depicting the quality of the terrestrial ecosystem (QTE) and assessing its changing triggers' contribution. Here, we summarized three major principles for selecting image sources in remote sensing data fusion. A continuous 30-m net vegetation productivity (NPP) dataset during 2000-2019 for the Guangdong-Hong Kong-Macao Greater Bay Area (GBA) was derived by using the Carnegie-Ames-Stanford approach model and pre-fused normalized difference vegetation index. The factors' contributions to the QTE changes were quantitatively assessed. The role of the QTE in affecting the socio-economic and its behind mechanisms was quantitatively investigated. The results showed that: (1) High-quality images sources are the preference for spatio-temporal fusion of remote sensing data. Images with close month, the same season and year, and sensors should be then selected. Images of different sensors with similar spectral bandwidth, the ones from adjacent years and seasons, can be alternately considered. (2) Fine-resolution NPP has higher accuracy than coarse-resolution NPP and has marked advantages in finely characterizing the QTE. In the past 20 years, the QTE in the GBA has shown a fluctuating increasing trend (0.20 Tg C/yr). (3) Human activities contributed 54.19% of the QTE changes in the GBA, and dominates the QTE changes in the central rapidly urbanizing areas. Residual factors accounted for an overall contribution ratio of 35.71%. Climate change dominants the peripheral forest variations in the GBA. (4) In the GBA, the improvement of QTE has a significant positive socio-economic impact, it contributes to the GDP increment firstly then the GDP aggregate indirectly. Our results highlight that it is of great urgent to estimate long-term continuous NPP with high spatio-temporal resolution globally. Controlling strategies should be implemented to reduce factitious impacts on QTE. High level of ecological and environmental protection promotes the sustainable development.

High-Spatial-Resolution NDVI Reconstruction with GA-ANN

The normalized differential vegetation index (NDVI) for Landsat is not continuous on the time scale due to the long revisit period and the influence of clouds and cloud shadows, such that the Landsat NDVI needs to be filled in and reconstructed. This study proposed a method based on the genetic algorithm-artificial neural network (GA-ANN) algorithm to reconstruct the Landsat NDVI when it has been affected by clouds, cloud shadows, and uncovered areas by relying on the MODIS characteristics for a wide coverage area. According to the self-validating results of the model test, the RMSE, MAE, and R were 0.0508, 0.0557, and 0.8971, respectively. Compared with the existing research, the reconstruction model based on the GA-ANN algorithm achieved a higher precision than the enhanced spatial and temporal adaptive reflectance fusion model (ESTARFM) and the flexible space-time data fusion algorithm (FSDAF) for complex land use types. The reconstructed method based on the GA-ANN algorithm had a higher root mean square error (RMSE) and mean absolute error (MAE). Then, the Sentinel NDVI data were used to verify the accuracy of the results. The validation results showed that the reconstruction method was superior to other methods in the sample plots with complex land use types. Especially on the time scale, the obtained NDVI results had a strong correlation with the Sentinel NDVI data. The correlation coefficient (R) of the GA-ANN algorithm reconstruction's NDVI and the Sentinel NDVI data was more than 0.97 for the land use types of cropland, forest, and grassland. Therefore, the reconstruction model based on the GA-ANN algorithm could effectively fill in the clouds, cloud shadows, and uncovered areas, and produce NDVI long-series data with a high spatial resolution.

An Improved Spatiotemporal Data Fusion Method for Snow-Covered Mountain Areas Using Snow Index and Elevation Information

Remote sensing images with high spatial and temporal resolution in snow-covered areas are important for forecasting avalanches and studying the local weather. However, it is difficult to obtain images with high spatial and temporal resolution by a single sensor due to the limitations of technology and atmospheric conditions. The enhanced spatial and temporal adaptive reflectance fusion model (ESTARFM) can fill in the time-series gap of remote sensing images, and it is widely used in spatiotemporal fusion. However, this method cannot accurately predict the change when there is a change in surface types. For example, a snow-covered surface will be revealed as the snow melts, or the surface will be covered with snow as snow falls. These sudden changes in surface type may not be predicted by this method. Thus, this study develops an improved spatiotemporal method ESTARFM (iESTARFM) for the snow-covered mountain areas in Nepal by introducing NDSI and DEM information to simulate the snow-covered change to improve the accuracy of selecting similar pixels. Firstly, the change in snow cover is simulated according to NDSI and DEM. Then, similar pixels are selected according to the change in snow cover. Finally, NDSI is added to calculate the weights to predict the pixels at the target time. Experimental results show that iESTARFM can reduce the bright abnormal patches in the land area compared to ESTARFM. For spectral accuracy, iESTARFM performs better than ESTARFM with the root mean square error (RMSE) being reduced by 0.017, the correlation coefficient (r) being increased by 0.013, and the Structural Similarity Index Measure (SSIM) being increased by 0.013. For spatial accuracy, iESTARFM can generate clearer textures, with Robert's edge (Edge) being reduced by 0.026. These results indicate that iESTARFM can obtain higher prediction results and maintain more spatial details, which can be used to generate dense time series images for snow-covered mountain areas.

Hierarchical Classification of Soybean in the Brazilian Savanna Based on Harmonized Landsat Sentinel Data

The Brazilian Savanna presents a complex agricultural dynamic and cloud cover issues; therefore, there is a need for new strategies for more detailed agricultural monitoring. Using a hierarchical classification system, we explored the Harmonized Landsat Sentinel-2 (HLS) dataset to detect soybean in western Bahia, Brazil. Multispectral bands (MS) and vegetation indices (VIs) from October 2021 to March 2022 were used as variables to feed Random Forest models, and the performances of the complete HLS time-series, HLSS30 (harmonized Sentinel), HLSL30 (harmonized Landsat), and Landsat 8 OLI (L8) were compared. At Level 1 (agricultural areas × native vegetation), HLS, HLSS30, and L8 produced identical models using MS + VIs, with 0.959 overall accuracies (OA) and Kappa of 0.917. At Level 2 (annual crops × perennial crops × pasturelands), HLS and L8 achieved an OA of 0.935 and Kappa > 0.89 using only VIs. At Level 3 (soybean × other annual crops), the HLS MS + VIs model achieved the best performance, with OA of 0.913 and Kappa of 0.808. Our results demonstrated the potential of the new HLS dataset for medium-resolution mapping initiatives at the crop level, which can impact decision-making processes involving large-scale soybean production and agricultural sustainability.

A New Method for Calculating Water Quality Parameters by Integrating Space–Ground Hyperspectral Data and Spectral-In Situ Assay Data

The effective integration of aerial remote sensing data and ground multi-source data has always been one of the difficulties of quantitative remote sensing. A new monitoring mode is designed, which installs the hyperspectral imager on the UAV and places a buoy spectrometer on the river. Water samples are collected simultaneously to obtain in situ assay data of total phosphorus, total nitrogen, COD, turbidity, and chlorophyll during data collection. The cross-correlogram spectral matching (CCSM) algorithm is used to match the data of the buoy spectrometer with the UAV spectral data to significantly reduce the UAV data noise. An absorption characteristics recognition algorithm (ACR) is designed to realize a new method for comparing UAV data with laboratory data. This method takes into account the spectral characteristics and the correlation characteristics of test data synchronously. It is concluded that the most accurate water quality parameters can be calculated by using the regression method under five scales after the regression tests of the multiple linear regression method (MLR), support vector machine method (SVM), and neural network (NN) method. This new working mode of integrating spectral imager data with point spectrometer data will become a trend in water quality monitoring.

Applying a Wavelet Transform Technique to Optimize General Fitting Models for SM Analysis: A Case Study in Downscaling over the Qinghai–Tibet Plateau

Soil moisture (SM) is an important land-surface parameter. Although microwave remote sensing is recognized as one of the most appropriate methods for retrieving SM, such retrievals often cannot meet the requirements of specific applications because of their coarse spatial resolution and spatiotemporal data gaps. A range of general models (GMs) for SM analysis topics (e.g., gap-filling, forecasting, and downscaling) have been introduced to address these shortcomings. This work presents a novel strategy (i.e., optimized wavelet-coupled fitting method (OWCM)) to enhance the fitting accuracy of GMs by introducing a wavelet transform (WT) technique. Four separate GMs are selected, i.e., elastic network regression, area-to-area regression kriging, random forest regression, and neural network regression. The fitting procedures are then tested within a downscaling analysis implemented between aggregated Global Land Surface Satellite products (i.e., LAI, FVC, albedo), Thermal and Reanalysis Integrating Medium-resolution Spatial-seamless LST, and Random Forest Soil Moisture (RFSM) datasets in both the WT space and the regular space. Then, eight fine-resolution SM datasets mapped from the trained GMs and OWCMs are analyzed using direct comparisons with in situ SM measurements and indirect intercomparisons between the aggregated OWCM-/GM-derived SM and RFSM. The results demonstrate that OWCM-derived SM products are generally closer to the in situ SM observations, and better capture in situ SM dynamics during the unfrozen season, compared to the corresponding GM-derived SM product, which shows fewer time changes and more stable trends. Moreover, OWCM-derived SM products represent a significant improvement over corresponding GM-derived SM products in terms of their ability to spatially and temporally match RFSM. Although spatial heterogeneity still substantially impacts the fitting accuracies of both GM and OWCM SM products, the improvements of OWCMs over GMs are significant. This improvement can likely be attributed to the fitting procedure of OWCMs implemented in the WT space, which better captures high- and low-frequency image features than in the regular space.

STF-EGFA: A Remote Sensing Spatiotemporal Fusion Network with Edge-Guided Feature Attention

Spatiotemporal fusion in remote sensing plays an important role in Earth science applications by using information complementarity between different remote sensing data to improve image performance. However, several problems still exist, such as edge contour blurring and uneven pixels between the predicted image and the real ground image, in the extraction of salient features by convolutional neural networks (CNNs). We propose a spatiotemporal fusion method with edge-guided feature attention based on remote sensing, called STF-EGFA. First, an edge extraction module is used to maintain edge details, which effectively solves the boundary blurring problem. Second, a feature fusion attention module is used to make adaptive adjustments to the extracted features. Among them, the spatial attention mechanism is used to solve the problem of weight variation in different channels of the network. Additionally, the problem of uneven pixel distribution is addressed with a pixel attention (PA) mechanism to highlight the salient features. We transmit the different features extracted by the edge module and the encoder to the feature attention (FA) module at the same time after the union. Furthermore, the weights of edges, pixels, channels and other features are adaptively learned. Finally, three remote sensing spatiotemporal fusion datasets, Ar Horqin Banner (AHB), Daxing and Tianjin, are used to verify the method. Experiments proved that the proposed method outperformed three typical comparison methods in terms of the overall visual effect and five objective evaluation indexes: spectral angle mapper (SAM), peak signal-to-noise ratio (PSNR), spatial correlation coefficient (SCC), structural similarity (SSIM) and root mean square error (RMSE). Thus, the proposed spatiotemporal fusion algorithm is feasible for remote sensing analysis.

Quantitative Evaluation of Grassland SOS Estimation Accuracy Based on Different MODIS-Landsat Spatio-Temporal Fusion Datasets

Estimating the Start of Growing Season (SOS) of grassland on the global scale is an important scientific issue since it can reflect the response of the terrestrial ecosystem to environmental changes and determine the start time of grazing. However, most remote sensing data has coarse- temporal and spatial resolution, resulting in low accuracy of SOS retrieval based on remote sensing methods. In recent years, much research has focused on multi-source data fusion technology to improve the spatio-temporal resolution of remote sensing information, and to provide a feasible path for high-accuracy remote sensing inversion of SOS. Nevertheless, there is still a lack of quantitative evaluation for the accuracy of these data fusion methods in SOS estimation. Therefore, in this study, the SOS estimation accuracy is quantitatively evaluated based on the spatio-temporal fusion daily datasets through the Spatial and Temporal Adaptive Reflectance Fusion Model (STARFM) and other models in Xilinhot City, Inner Mongolia, China. The results show that: (1) the accuracy of SOS estimation based on spatio-temporal fusion daily datasets has been slightly improved, the average Root Mean Square Error (RMSE) of SOS based on 8d composite datasets is 11.1d, and the best is 9.7d (fstarfm8); (2) the estimation accuracy based on 8d composite datasets (RMSE¯ = 11.1d) is better than daily fusion datasets (RMSE¯ = 18.2d); (3) the lack of the Landsat data during the SOS would decrease the quality of the fusion datasets, which ultimately reduces the accuracy of the SOS estimation. The RMSE¯ of SOS based on all three models increases by 11.1d, and the STARFM is least affected, just increases 2.7d. The results highlight the potential of the spatio-temporal data fusion method in high-accuracy grassland SOS estimation. It also shows that the dataset fused by the STARFM algorithm and composed for 8 days is better for SOS estimation.

Remote Sensing and Field Survey Data Integration to Investigate on the Evolution of the Coastal Area: The Case Study of Bagnara Calabra (Southern Italy)

Coastal areas worldwide are the result of a weak balance between man and the natural environment. They are exposed to strong anthropogenic pressure and natural hazard events whose intensity has increased in recent decades. In this frame, the satellite and drone monitoring systems as well as field survey are key tools to learn about the factors responsible for coastal changes. Here we describe the formation and dismantling of a fan delta at Sfalassà Stream mouth, Calabria Region (Southern Italy) to shed light on the environmental drivers modelling this coast. The flood event of 2 November 2015 placed approximately 25,000 m3 of coarse sand and gravel sediments in a few hours forming a fan-shaped delta, while three main storm surges, occurring from November 2015 to January 2016, caused its dismantling. Sentinel 2 images and several photographs captured the gradual erosion of fan delta highlighting its complete dismantling in about 3 months. The eroded sediments only partially feed the neighbouring beaches, as they were rapidly funnelled several hundred metres seaward by submarine channels whose heads cut back up at depths <10 m. This analysis showed that observing systems with high spatial and temporal resolution provide the proper knowledge to model the processes that characterise this transitional environment. They are fundamental tools for coastal zone management, which aims to ensure the sustainability of coastal zones by mitigating the effects of erosion and flooding.

Impact of the Dates of Input Image Pairs on Spatio-Temporal Fusion for Time Series with Different Temporal Variation Patterns

Dense time series of remote sensing images with high spatio-temporal resolution are critical for monitoring land surface dynamics in heterogeneous landscapes. Spatio-temporal fusion is an effective solution to obtaining such time series images. Many spatio-temporal fusion methods have been developed for producing high spatial resolution images at frequent intervals by blending fine spatial images and coarse spatial resolution images. Previous studies have revealed that the accuracy of fused images depends not only on the fusion algorithm, but also on the input image pairs being used. However, the impact of input images dates on the fusion accuracy for time series with different temporal variation patterns remains unknown. In this paper, the impact of input image pairs on the fusion accuracy for monotonic linear change (MLC), monotonic non-linear change (MNLC), and non-monotonic change (NMC) time periods were evaluated, respectively, and the optimal selection strategies of input image dates for different situations were proposed. The 16-day composited NDVI time series (i.e., Collection 6 MODIS NDVI product) were used to present the temporal variation patterns of land surfaces in the study areas. To obtain sufficient observation dates to evaluate the impact of input image pairs on the spatio-temporal fusion accuracy, we utilized the Harmonized Landsat-8 Sentinel-2 (HLS) data. The ESTARFM was selected as the spatio-temporal fusion method for this study. The results show that the impact of input image date on the accuracy of spatio-temporal fusion varies with the temporal variation patterns of the time periods being fused. For the MLC period, the fusion accuracy at the prediction date (PD) is linearly correlated to the time interval between the change date (CD) of the input image and the PD, but the impact of the input image date on the fusion accuracy at the PD is not very significant. For the MNLC period, the fusion accuracy at the PD is non-linearly correlated to the time interval between the CD and the PD, the impact of the time interval between the CD and the PD on the fusion accuracy is more significant for the MNLC than for the MLC periods. Given the similar change of time intervals between the CD and the PD, the increments of R2 of fusion result for the MNLC is over ten times larger than those for the MLC. For the NMC period, a shorter time interval between the CD and the PD does not lead to higher fusion accuracies. On the contrary, it may lower the fusion accuracy. This study suggests that temporal variation patterns of the data must be taken into account when selecting optimal dates of input images in the fusion model.

Improved Daily Evapotranspiration Estimation Using Remotely Sensed Data in a Data Fusion System

Evapotranspiration (ET) represents crop water use and is a key indicator of crop health. Accurate estimation of ET is critical for agricultural irrigation and water resource management. ET retrieval using energy balance methods with remotely sensed thermal infrared data as the key input has been widely applied for irrigation scheduling, yield prediction, drought monitoring and so on. However, limitations on the spatial and temporal resolution of available thermal satellite data combined with the effects of cloud contamination constrain the amount of detail that a single satellite can provide. Fusing satellite data from different satellites with varying spatial and temporal resolutions can provide a more continuous estimation of daily ET at field scale. In this study, we applied an ET fusion modeling system, which uses a surface energy balance model to retrieve ET using both Landsat and Moderate Resolution Imaging Spectroradiometer (MODIS) data and then fuses the Landsat and MODIS ET retrieval timeseries using the Spatial-Temporal Adaptive Reflectance Fusion Model (STARFM). In this paper, we compared different STARFM ET fusion implementation strategies over various crop lands in the central California. In particular, the use of single versus two Landsat-MODIS pair images to constrain the fusion is explored in cases of rapidly changing crop conditions, as in frequently harvested alfalfa fields, as well as an improved dual-pair method. The daily 30 m ET retrievals are evaluated with flux tower observations and analyzed based on land cover type. This study demonstrates improvement using the new dual-pair STARFM method compared with the standard one-pair STARFM method in estimating daily field scale ET for all the major crop types in the study area.

Spatiotemporal Fusion Modelling Using STARFM: Examples of Landsat 8 and Sentinel-2 NDVI in Bavaria

The increasing availability and variety of global satellite products provide a new level of data with different spatial, temporal, and spectral resolutions; however, identifying the most suited resolution for a specific application consumes increasingly more time and computation effort. The region’s cloud coverage additionally influences the choice of the best trade-off between spatial and temporal resolution, and different pixel sizes of remote sensing (RS) data may hinder the accurate monitoring of different land cover (LC) classes such as agriculture, forest, grassland, water, urban, and natural-seminatural. To investigate the importance of RS data for these LC classes, the present study fuses NDVIs of two high spatial resolution data (high pair) (Landsat (30 m, 16 days; L) and Sentinel-2 (10 m, 5–6 days; S), with four low spatial resolution data (low pair) (MOD13Q1 (250 m, 16 days), MCD43A4 (500 m, one day), MOD09GQ (250 m, one-day), and MOD09Q1 (250 m, eight day)) using the spatial and temporal adaptive reflectance fusion model (STARFM), which fills regions’ cloud or shadow gaps without losing spatial information. These eight synthetic NDVI STARFM products (2: high pair multiply 4: low pair) offer a spatial resolution of 10 or 30 m and temporal resolution of 1, 8, or 16 days for the entire state of Bavaria (Germany) in 2019. Due to their higher revisit frequency and more cloud and shadow-free scenes (S = 13, L = 9), Sentinel-2 (overall R2 = 0.71, and RMSE = 0.11) synthetic NDVI products provide more accurate results than Landsat (overall R2 = 0.61, and RMSE = 0.13). Likewise, for the agriculture class, synthetic products obtained using Sentinel-2 resulted in higher accuracy than Landsat except for L-MOD13Q1 (R2 = 0.62, RMSE = 0.11), resulting in similar accuracy preciseness as S-MOD13Q1 (R2 = 0.68, RMSE = 0.13). Similarly, comparing L-MOD13Q1 (R2 = 0.60, RMSE = 0.05) and S-MOD13Q1 (R2 = 0.52, RMSE = 0.09) for the forest class, the former resulted in higher accuracy and precision than the latter. Conclusively, both L-MOD13Q1 and S-MOD13Q1 are suitable for agricultural and forest monitoring; however, the spatial resolution of 30 m and low storage capacity makes L-MOD13Q1 more prominent and faster than that of S-MOD13Q1 with the 10-m spatial resolution.

A Spatiotemporal Fusion Method Based on Multiscale Feature Extraction and Spatial Channel Attention Mechanism

Remote sensing satellite images with a high spatial and temporal resolution play a crucial role in Earth science applications. However, due to technology and cost constraints, it is difficult for a single satellite to achieve both a high spatial resolution and high temporal resolution. The spatiotemporal fusion method is a cost-effective solution for generating a dense temporal data resolution with a high spatial resolution. In recent years, spatiotemporal image fusion based on deep learning has received wide attention. In this article, a spatiotemporal fusion method based on multiscale feature extraction and a spatial channel attention mechanism is proposed. Firstly, the method uses a multiscale mechanism to fully utilize the structural features in the images. Then a novel attention mechanism is used to capture both spatial and channel information; finally, the rich features and spatial and channel information are used to fuse the images. Experimental results obtained from two datasets show that the proposed method outperforms existing fusion methods in both subjective and objective evaluations.

Evaluation of Random Forests (RF) for Regional and Local-Scale Wheat Yield Prediction in Southeast Australia

Wheat accounts for more than 50% of Australia's total grain production. The capability to generate accurate in-season yield predictions is important across all components of the agricultural value chain. The literature on wheat yield prediction has motivated the need for more novel works evaluating machine learning techniques such as random forests (RF) at multiple scales. This research applied a Random Forest Regression (RFR) technique to build regional and local-scale yield prediction models at the pixel level for three southeast Australian wheat-growing paddocks, each located in Victoria (VIC), New South Wales (NSW) and South Australia (SA) using 2018 yield maps from data supplied by collaborating farmers. Time-series Normalized Difference Vegetation Index (NDVI) data derived from Planet's high spatio-temporal resolution imagery, meteorological variables and yield data were used to train, test and validate the models at pixel level using Python libraries for (a) regional-scale three-paddock composite and (b) individual paddocks. The composite region-wide RF model prediction for the three paddocks performed well (R² = 0.86, RMSE = 0.18 t ha^-1). RF models for individual paddocks in VIC (R² = 0.89, RMSE = 0.15 t ha^-1) and NSW (R² = 0.87, RMSE = 0.07 t ha^-1) performed well, but moderate performance was seen for SA (R² = 0.45, RMSE = 0.25 t ha^-1). Generally, high values were underpredicted and low values overpredicted. This study demonstrated the feasibility of applying RF modeling on satellite imagery and yielded 'big data' for regional as well as local-scale yield prediction.

Evaluation of Landsat 8-like Land Surface Temperature by Fusing Landsat 8 and MODIS Land Surface Temperature Product

High-spatiotemporal-resolution land surface temperature (LST) is a crucial parameter in various environmental monitoring. However, due to the limitation of sensor trade-off between the spatial and temporal resolutions, such data are still unavailable. Therefore, the generation and verification of such data are of great value. The spatiotemporal fusion algorithm, which can be used to improve the spatiotemporal resolution, is widely used in Landsat and MODIS data to generate Landsat-like images, but there is less exploration of combining long-time series MODIS LST and Landsat 8 LST product to generate Landsat 8-like LST. The purpose of this study is to evaluate the accuracy of the long-time series Landsat 8 LST product and the Landsat 8-like LST generated by spatiotemporal fusion. In this study, based on the Landsat 8 LST product and MODIS LST product, Landsat 8-like LST is generated using Spatial and Temporal Adaptive Reflectance Fusion Model (STARFM), Enhanced STARFM (ESTARFM), and the Flexible Spatiotemporal DAta Fusion (FSDAF) algorithm, and tested and verified in the research area located in Gansu Province, China. In this process, Landsat 8 LST product was verified based on ground measurements, and the fusion results were comprehensively evaluated based on ground measurements and actual Landsat 8 LST images. Ground measurements verification indicated that Landsat 8 LST product was highly consistent with ground measurements. The Root Mean Square Error (RMSE) was 2.862 K, and the coefficient of determination R2 was 0.952 at All stations. Good fusion results can be obtained for the three spatiotemporal algorithms, and the ground measurements verified at All stations show that R2 was more significant than 0.911. ESTARFM had the best fusion result (R2 = 0.915, RMSE = 3.661 K), which was better than STARFM (R2 = 0.911, RMSE = 3.746 K) and FSDAF (R2 = 0.912, RMSE = 3.786 K). Based on the actual Landsat 8 LST images verification, the fusion images were highly consistent with actual Landsat 8 LST images. The average RMSE of fusion images about STARFM, ESTARFM, and FSDAF were 2.608 K, 2.245 K, and 2.565 K, respectively, and ESTARFM is better than STARFM and FSDAF in most cases. Combining the above verification, the fusion results of the three algorithms were reliable and ESTARFM had the highest fusion accuracy.

Towards Routine Mapping of Crop Emergence within the Season Using the Harmonized Landsat and Sentinel-2 Dataset

Crop emergence is a critical stage for crop development modeling, crop condition monitoring, and biomass accumulation estimation. Green-up dates (or the start of the season) detected from remote sensing time series are related to, but generally lag, crop emergence dates. In this paper, we refine the within-season emergence (WISE) algorithm and extend application to five Corn Belt states (Iowa, Illinois, Indiana, Minnesota, and Nebraska) using routine harmonized Landsat and Sentinel-2 (HLS) data from 2018 to 2020. Green-up dates detected from the HLS time series were assessed using field observations and near-surface measurements from PhenoCams. Statistical descriptions of green-up dates for corn and soybeans were generated and compared to county-level planting dates and district- to state-level crop emergence dates reported by the National Agricultural Statistics Service (NASS). Results show that emergence dates for corn and soybean can be reliably detected within the season using the HLS time series acquired during the early growing season. Compared to observed crop emergence dates, green-up dates from HLS using WISE were ~3 days later at the field scale (30-m). The mean absolute difference (MAD) was ~7 days and the root mean square error (RMSE) was ~9 days. At the state level, the mean differences between median HLS green-up date and median crop emergence date were within 2 days for 2018–2020. At this scale, MAD was within 4 days, and RMSE was less than 5 days for both corn and soybeans. The R-squares were 0.73 and 0.87 for corn and soybean, respectively. The 2019 late emergence of crops in Corn Belt states (1–4 weeks to five-year average) was captured by HLS green-up date retrievals. This study demonstrates that routine within-season mapping of crop emergence/green-up at the field scale is practicable over large regions using operational satellite data. The green-up map derived from HLS during the growing season provides valuable information on spatial and temporal variability in crop emergence that can be used for crop monitoring and refining agricultural statistics used in broad-scale modeling efforts.

Reducing Scaling Effect on Downscaled Land Surface Temperature Maps in Heterogenous Urban Environments

The literature review indicates that a scaling effect does exist in downscaling land surface temperature (DLST) processes, and no substantial methods were specially developed for addressing it. In this research, the main aim is to develop a new method to reduce the scaling effect on DLST maps at high resolutions. A thermal component-based thermal spectral unmixing (TSU) model was modified and a multiple regression (REG) model was adopted to create DLST maps at high resolutions. A combined variance of red and NIR bands at a very high resolution with a difference image between upscaled LST and DLST was used to develop a new method. With two case data sets, LSTs at coarse resolutions were downscaled by using the modified TSU model and the REG model to create DLST results. The new method with a correction term expression (a linear model created by using a semi-empirical approach) was used to improve the DLST maps in the two case study areas. The experimental results indicate that the new method could reduce the root mean square error and the mean absolute error >30% and >33%, respectively, and thus demonstrate that the proposed method was effective and significant, especially reducing the scaling effect on DLST results at very high resolutions. The novel significance for the new method is directly reducing the scaling effect on DLST maps at high resolutions.

A Robust Hybrid Deep Learning Model for Spatiotemporal Image Fusion

Dense time-series remote sensing data with detailed spatial information are highly desired for the monitoring of dynamic earth systems. Due to the sensor tradeoff, most remote sensing systems cannot provide images with both high spatial and temporal resolutions. Spatiotemporal image fusion models provide a feasible solution to generate such a type of satellite imagery, yet existing fusion methods are limited in predicting rapid and/or transient phenological changes. Additionally, a systematic approach to assessing and understanding how varying levels of temporal phenological changes affect fusion results is lacking in spatiotemporal fusion research. The objective of this study is to develop an innovative hybrid deep learning model that can effectively and robustly fuse the satellite imagery of various spatial and temporal resolutions. The proposed model integrates two types of network models: super-resolution convolutional neural network (SRCNN) and long short-term memory (LSTM). SRCNN can enhance the coarse images by restoring degraded spatial details, while LSTM can learn and extract the temporal changing patterns from the time-series images. To systematically assess the effects of varying levels of phenological changes, we identify image phenological transition dates and design three temporal phenological change scenarios representing rapid, moderate, and minimal phenological changes. The hybrid deep learning model, alongside three benchmark fusion models, is assessed in different scenarios of phenological changes. Results indicate the hybrid deep learning model yields significantly better results when rapid or moderate phenological changes are present. It holds great potential in generating high-quality time-series datasets of both high spatial and temporal resolutions, which can further benefit terrestrial system dynamic studies. The innovative approach to understanding phenological changes’ effect will help us better comprehend the strengths and weaknesses of current and future fusion models.

Extraction of Abandoned Land in Hilly Areas Based on the Spatio-Temporal Fusion of Multi-Source Remote Sensing Images

Hilly areas are important parts of the world’s landscape. A marginal phenomenon can be observed in some hilly areas, leading to serious land abandonment. Extracting the spatio-temporal distribution of abandoned land in such hilly areas can protect food security, improve people’s livelihoods, and serve as a tool for a rational land plan. However, mapping the distribution of abandoned land using a single type of remote sensing image is still challenging and problematic due to the fragmentation of such hilly areas and severe cloud pollution. In this study, a new approach by integrating Linear stretch (Ls), Maximum Value Composite (MVC), and Flexible Spatiotemporal DAta Fusion (FSDAF) was proposed to analyze the time-series changes and extract the spatial distribution of abandoned land. MOD09GA, MOD13Q1, and Sentinel-2 were selected as the basis of remote sensing images to fuse a monthly 10 m spatio-temporal data set. Three pieces of vegetation indices (VIs: ndvi, savi, ndwi) were utilized as the measures to identify the abandoned land. A multiple spatio-temporal scales sample database was established, and the Support Vector Machine (SVM) was used to extract abandoned land from cultivated land and woodland. The best extraction result with an overall accuracy of 88.1% was achieved by integrating Ls, MVC, and FSDAF, with the assistance of an SVM classifier. The fused VIs image set transcended the single source method (Sentinel-2) with greater accuracy by a margin of 10.8–23.6% for abandoned land extraction. On the other hand, VIs appeared to contribute positively to extract abandoned land from cultivated land and woodland. This study not only provides technical guidance for the quick acquirement of abandoned land distribution in hilly areas, but it also provides strong data support for the connection of targeted poverty alleviation to rural revitalization.

ERTFM: An Effective Model to Fuse Chinese GF-1 and MODIS Reflectance Data for Terrestrial Latent Heat Flux Estimation

Coarse spatial resolution sensors play a major role in capturing temporal variation, as satellite images that capture fine spatial scales have a relatively long revisit cycle. The trade-off between the revisit cycle and spatial resolution hinders the access of terrestrial latent heat flux (LE) data with both fine spatial and temporal resolution. In this paper, we firstly investigated the capability of an Extremely Randomized Trees Fusion Model (ERTFM) to reconstruct high spatiotemporal resolution reflectance data from a fusion of the Chinese GaoFen-1 (GF-1) and the Moderate Resolution Imaging Spectroradiometer (MODIS) products. Then, based on the merged reflectance data, we used a Modified-Satellite Priestley–Taylor (MS–PT) algorithm to generate LE products at high spatial and temporal resolutions. Our results illustrated that the ERTFM-based reflectance estimates showed close similarity with observed GF-1 images and the predicted NDVI agreed well with observed NDVI at two corresponding dates (r = 0.76 and 0.86, respectively). In comparison with other four fusion methods, including the widely used spatial and temporal adaptive reflectance fusion model (STARFM) and the enhanced STARFM, ERTFM had the best performance in terms of predicting reflectance (SSIM = 0.91; r = 0.77). Further analysis revealed that LE estimates using ERTFM-based data presented more detailed spatiotemporal characteristics and provided close agreement with site-level LE observations, with an R2 of 0.81 and an RMSE of 19.18 W/m2. Our findings suggest that the ERTFM can be used to improve LE estimation with high frequency and high spatial resolution, meaning that it has great potential to support agricultural monitoring and irrigation management.

Spatial Downscaling of Land Surface Temperature over Heterogeneous Regions Using Random Forest Regression Considering Spatial Features

Land surface temperature (LST) is one of the crucial parameters in the physical processes of the Earth. Acquiring LST images with high spatial and temporal resolutions is currently difficult because of the technical restriction of satellite thermal infrared sensors. Downscaling LST from coarse to fine spatial resolution is an effective means to alleviate this problem. A spatial random forest downscaling LST method (SRFD) was proposed in this study. Abundant predictor variables—including land surface reflection data, remote sensing spectral indexes, terrain factors, and land cover type data—were considered and applied for feature selection in SRFD. Moreover, the shortcoming of only focusing on information from point-to-point in previous statistics-based downscaling methods was supplemented by adding the spatial feature of LST. SRFD was applied to three different heterogeneous regions and compared with the results from three classical or excellent methods, including thermal image sharpening algorithm, multifactor geographically weighted regression, and random forest downscaling method. Results show that SRFD outperforms other methods in vision and statistics due to the benefits from the supplement of the LST spatial feature. Specifically, compared with RFD, the second-best method, the downscaling results of SRFD are 10% to 24% lower in root-mean-square error, 5% to 20% higher in the coefficient of determination, 11% to 25% lower in mean absolute error, and 4% to 17% higher in structural similarity index measure. Hence, we conclude that SRFD will be a promising LST downscaling method.

Computer Vision, IoT and Data Fusion for Crop Disease Detection Using Machine Learning: A Survey and Ongoing Research

Crop diseases constitute a serious issue in agriculture, affecting both quality and quantity of agriculture production. Disease control has been a research object in many scientific and technologic domains. Technological advances in sensors, data storage, computing resources and artificial intelligence have shown enormous potential to control diseases effectively. A growing body of literature recognizes the importance of using data from different types of sensors and machine learning approaches to build models for detection, prediction, analysis, assessment, etc. However, the increasing number and diversity of research studies requires a literature review for further developments and contributions in this area. This paper reviews state-of-the-art machine learning methods that use different data sources, applied to plant disease detection. It lists traditional and deep learning methods associated with the main data acquisition modalities, namely IoT, ground imaging, unmanned aerial vehicle imaging and satellite imaging. In addition, this study examines the role of data fusion for ongoing research in the context of disease detection. It highlights the advantage of intelligent data fusion techniques, from heterogeneous data sources, to improve plant health status prediction and presents the main challenges facing this field. The study concludes with a discussion of several current issues and research trends.

A long-term spatiotemporal analysis of biocrusts across a diverse arid environment: The case of the Israeli-Egyptian sandfield

Spatiotemporal data can be analyzed using spatial, time-series, and machine learning algorithms to extract regional biocrust trends. Analyzing the spatial trends of biocrusts through time, using satellite imagery, may improve the quantification and understanding of their change drivers. The current work strives to develop a unique framework for analyzing spatiotemporal trends of the spectral Crust Index (CI), thus identifying the drivers of the biocrusts' spatial and temporal patterns. To fulfill this goal, CI maps, derived from 31 annual Landsat images, were analyzed by applying advanced statistical and machine learning algorithms. A comprehensive overview of biocrusts' spatiotemporal patterns was achieved using an integrative approach, including a long-term analysis, using the Mann-Kendall (MK) statistical test, and a short-term analysis, using a rolling MK with a window size of five years. Additionally, temporal clustering, using the partition around medoids (PAM) algorithm, was applied to model the spatial multi-annual dynamics of the CI. A Granger Causality test was then applied to quantify the relations between CI dynamics and precipitation. The findings show that 88.7% of pixels experienced a significant negative change, and only 0.5% experienced a significant positive change. A strong association was found in temporal trends among all clusters (0.67 ≤ r ≤ 0.8), signifying a regional effect due to precipitation levels (p < 0.05 for most clusters). The biocrust dynamics were also locally affected by anthropogenic factors (0.58 > CI > 0.64 and 0.64 > CI > 0.71 for strongly and weakly affected regions, respectively). A spatiotemporal analysis of a series of spaceborne images may improve conservation management by evaluating biocrust development in drylands. The suggested framework may also by applied to various disciplines related to quantifying spatial and temporal trends.

Inferring Grassland Drought Stress with Unsupervised Learning from Airborne Hyperspectral VNIR Imagery

The 2018–2019 Central European drought had a grave impact on natural and managed ecosystems, affecting their health and productivity. We examined patterns in hyperspectral VNIR imagery using an unsupervised learning approach to improve ecosystem monitoring and the understanding of grassland drought responses. The main objectives of this study were (1) to evaluate the application of simplex volume maximisation (SiVM), an unsupervised learning method, for the detection of grassland drought stress in high-dimensional remote sensing data at the ecosystem scale and (2) to analyse the contributions of different spectral plant and soil traits to the computed stress signal. The drought status of the research site was assessed with a non-parametric standardised precipitation–evapotranspiration index (SPEI) and soil moisture measurements. We used airborne HySpex VNIR-1800 data from spring 2018 and 2019 to compare vegetation condition at the onset of the drought with the state after one year. SiVM, an interpretable matrix factorisation technique, was used to derive typical extreme spectra (archetypes) from the hyperspectral data. The classification of archetypes allowed for the inference of qualitative drought stress levels. The results were evaluated using a set of geophysical measurements and vegetation indices as proxy variables for drought-inhibited vegetation growth. The successful application of SiVM for grassland stress detection at the ecosystem canopy scale was verified in a correlation analysis. The predictor importance was assessed with boosted beta regression. In the resulting interannual stress model, carotenoid-related variables had among the highest coefficient values. The significance of the photochemical reflectance index that uses 512 nm as reference wavelength (PRI512) demonstrates the value of combining imaging spectrometry and unsupervised learning for the monitoring of vegetation stress. It also shows the potential of archetypical reflectance spectra to be used for the remote estimation of photosynthetic efficiency. More conclusive results could be achieved by using vegetation measurements instead of proxy variables for evaluation. It must also be investigated how the method can be generalised across ecosystems.

Deriving Non-Cloud Contaminated Sentinel-2 Images with RGB and Near-Infrared Bands from Sentinel-1 Images Based on a Conditional Generative Adversarial Network

Sentinel-2 images have been widely used in studying land surface phenomena and processes, but they inevitably suffer from cloud contamination. To solve this critical optical data availability issue, it is ideal to fuse Sentinel-1 and Sentinel-2 images to create fused, cloud-free Sentinel-2-like images for facilitating land surface applications. In this paper, we propose a new data fusion model, the Multi-channels Conditional Generative Adversarial Network (MCcGAN), based on the conditional generative adversarial network, which is able to convert images from Domain A to Domain B. With the model, we were able to generate fused, cloud-free Sentinel-2-like images for a target date by using a pair of reference Sentinel-1/Sentinel-2 images and target-date Sentinel-1 images as inputs. In order to demonstrate the superiority of our method, we also compared it with other state-of-the-art methods using the same data. To make the evaluation more objective and reliable, we calculated the root-mean-square-error (RSME), R2, Kling–Gupta efficiency (KGE), structural similarity index (SSIM), spectral angle mapper (SAM), and peak signal-to-noise ratio (PSNR) of the simulated Sentinel-2 images generated by different methods. The results show that the simulated Sentinel-2 images generated by the MCcGAN have a higher quality and accuracy than those produced via the previous methods.

Resolution Enhancement of Remotely Sensed Land Surface Temperature: Current Status and Perspectives

Remotely sensed land surface temperature (LST) distribution has played a valuable role in land surface processes studies from local to global scales. However, it is still difficult to acquire concurrently high spatiotemporal resolution LST data due to the trade-off between spatial and temporal resolutions in thermal remote sensing. To address this problem, various methods have been proposed to enhance the resolutions of LST data, and substantial progress in this field has been achieved in recent years. Therefore, this study reviewed the current status of resolution enhancement methods for LST data. First, three groups of enhancement methods—spatial resolution enhancement, temporal resolution enhancement, and simultaneous spatiotemporal resolution enhancement—were comprehensively investigated and analyzed. Then, the quality assessment strategies for LST resolution enhancement methods and their advantages and disadvantages were specifically discussed. Finally, key directions for future studies in this field were suggested, i.e., synergy between process-driven and data-driven methods, cross-comparison among different methods, and improvement in localization strategy.

Research Gap Analysis of Remote Sensing Application in Fisheries: Prospects for Achieving the Sustainable Development Goals

Remote sensing (RS) technology, which can facilitate the sustainable management and development of fisheries, is easily accessible and exhibits high performance. It only requires the collection of sufficient information, establishment of databases and input of human and capital resources for analysis. However, many countries are unable to effectively ensure the sustainable development of marine fisheries due to technological limitations. The main challenge is the gap in the conditions for sustainable development between developed and developing countries. Therefore, this study applied the Web of Science database and geographic information systems to analyze the gaps in fisheries science in various countries over the past 10 years. Most studies have been conducted in the offshore marine areas of the northeastern United States of America. In addition, all research hotspots were located in the Northern Hemisphere, indicating a lack of relevant studies from the Southern Hemisphere. This study also found that research hotspots of satellite RS applications in fisheries were mainly conducted in (1) the northeastern sea area in the United States, (2) the high seas area of the North Atlantic Ocean, (3) the surrounding sea areas of France, Spain and Portugal, (4) the surrounding areas of the Indian Ocean and (5) the East China Sea, Yellow Sea and Bohai Bay sea areas to the north of Taiwan. A comparison of publications examining the three major oceans indicated that the Atlantic Ocean was the most extensively studied in terms of RS applications in fisheries, followed by the Indian Ocean, while the Pacific Ocean was less studied than the aforementioned two regions. In addition, all research hotspots were located in the Northern Hemisphere, indicating a lack of relevant studies from the Southern Hemisphere. The Atlantic Ocean and the Indian Ocean have been the subjects of many local in-depth studies; in the Pacific Ocean, the coastal areas have been abundantly investigated, while offshore local areas have only been sporadically addressed. Collaboration and partnership constitute an efficient approach for transferring skills and technology across countries. For the achievement of the sustainable development goals (SDGs) by 2030, research networks can be expanded to mitigate the research gaps and improve the sustainability of marine fisheries resources.

A Hybrid Deep Learning-Based Spatiotemporal Fusion Method for Combining Satellite Images with Different Resolutions

Spatiotemporal fusion (STF) is considered a feasible and cost-effective way to deal with the trade-off between the spatial and temporal resolution of satellite sensors, and to generate satellite images with high spatial and high temporal resolutions. This is achieved by fusing two types of satellite images, i.e., images with fine temporal but rough spatial resolution, and images with fine spatial but rough temporal resolution. Numerous STF methods have been proposed, however, it is still a challenge to predict both abrupt landcover change, and phenological change, accurately. Meanwhile, robustness to radiation differences between multi-source satellite images is crucial for the effective application of STF methods. Aiming to solve the abovementioned problems, in this paper we propose a hybrid deep learning-based STF method (HDLSFM). The method formulates a hybrid framework for robust fusion with phenological and landcover change information with minimal input requirements, and in which a nonlinear deep learning-based relative radiometric normalization, a deep learning-based superresolution, and a linear-based fusion are combined to address radiation differences between different types of satellite images, landcover, and phenological change prediction. Four comparative experiments using three popular STF methods, i.e., spatial and temporal adaptive reflectance fusion model (STARFM), flexible spatiotemporal data fusion (FSDAF), and Fit-FC, as benchmarks demonstrated the effectiveness of the HDLSFM in predicting phenological and landcover change. Meanwhile, HDLSFM is robust for radiation differences between different types of satellite images and the time interval between the prediction and base dates, which ensures its effectiveness in the generation of fused time-series data.

Spatiotemporal Fusion of Formosat-2 and Landsat-8 Satellite Images: A Comparison of “Super Resolution-Then-Blend” and “Blend-Then-Super Resolution” Approaches

The spatiotemporal fusion technique has the advantages of generating time-series images with high-spatial and high-temporal resolution from coarse-resolution to fine-resolution images. A hybrid fusion method that integrates image blending (i.e., spatial and temporal adaptive reflectance fusion model, STARFM) and super-resolution (i.e., very deep super resolution, VDSR) techniques for the spatiotemporal fusion of 8 m Formosat-2 and 30 m Landsat-8 satellite images is proposed. Two different fusion approaches, namely Blend-then-Super-Resolution and Super-Resolution (SR)-then-Blend, were developed to improve the results of spatiotemporal fusion. The SR-then-Blend approach performs SR before image blending. The SR refines the image resampling stage on generating the same pixel-size of coarse- and fine-resolution images. The Blend-then-SR approach is aimed at refining the spatial details after image blending. Several quality indices were used to analyze the quality of the different fusion approaches. Experimental results showed that the performance of the hybrid method is slightly better than the traditional approach. Images obtained using SR-then-Blend are more similar to the real observed images compared with images acquired using Blend-then-SR. The overall mean bias of SR-then-Blend was 4% lower than Blend-then-SR, and nearly 3% improvement for overall standard deviation in SR-B. The VDSR technique reduces the systematic deviation in spectral band between Formosat-2 and Landsat-8 satellite images. The integration of STARFM and the VDSR model is useful for improving the quality of spatiotemporal fusion.

Gap Filling for Historical Landsat NDVI Time Series by Integrating Climate Data

High-quality Normalized Difference Vegetation Index (NDVI) time series are essential in studying vegetation phenology, dynamic monitoring, and global change. Gap filling is the most important issue in reconstructing NDVI time series from satellites with high spatial resolution, e.g., the Landsat series and Chinese GaoFen-1/6 series. Due to the sparse revisit frequencies of high-resolution satellites, traditional reconstruction approaches face the challenge of dealing with large gaps in raw NDVI time series data. In this paper, a climate incorporated gap-filling (CGF) method is proposed for the reconstruction of Landsat historical NDVI time series data. The CGF model considers the relationship of the NDVI time series and climate conditions between two adjacent years. Climate variables, including downward solar shortwave radiation, precipitation, and temperature, are used to characterize the constrain factors of vegetation growth. Radial basis function networks (RBFNs) are used to link the NDVI time series between two adjacent years with variabilities in climatic conditions. An RBFN predicted a background NDVI time series in the target year, and the observed NDVI values in this year were used to adjust the predicted NDVI time series. Finally, the NDVI time series were recursively reconstructed from 2018 to 1986. The experiments were performed in a heterogeneous region in the Qilian Mountains. The results demonstrate that the proposed method can accurately reconstruct and generate continuous 30 m 8-day NDVI time series using Landsat observations. The CGF method outperforms traditional time series reconstruction methods (e.g., the harmonic analysis of time series (HANTS) and Savitzky-Golay (SG) filter methods) when the raw time series is contaminated with large gaps, which widely exist in Landsat images.

Assessing the Accuracy of Landsat-MODIS NDVI Fusion with Limited Input Data: A Strategy for Base Data Selection

Despite its wide applications, the spatiotemporal fusion of coarse- and fine-resolution satellite images is limited primarily to the availability of clear-sky fine-resolution images, which are commonly scarce due to unfavorable weather, and such a limitation might cause errors in spatiotemporal fusion. Thus, the effective use of limited fine-resolution images, while critical, remains challenging. To address this issue, in this paper we propose a new phenological similarity strategy (PSS) to select the optimal combination of image pairs for a prediction date. The PSS considers the temporal proximity and phenological similarity between the base and prediction images and computes a weight for identifying the optimal combination of image pairs. Using the PSS, we further evaluate the influence of input data on the fusion accuracy by varying the number and temporal distribution of input images. The results show that the PSS (mean R = 0.827 and 0.760) outperforms the nearest date (mean R = 0.786 and 0.742) and highest correlation (mean R = 0.821 and 0.727) strategies in both the enhanced spatial and temporal adaptive reflectance fusion model (ESTARFM) and the linear mixing growth model (LMGM), respectively, for fusing Landsat 8 OLI and MODIS NDVI datasets. Furthermore, base images adequately covering different growth stages yield better predictability than simply increasing the number of base images.

Reconstruction of High-Temporal- and High-Spatial-Resolution Reflectance Datasets Using Difference Construction and Bayesian Unmixing

High-temporal- and high-spatial-resolution reflectance datasets play a vital role in monitoring dynamic changes at the Earth’s land surface. So far, many sensors have been designed with a trade-off between swath width and pixel size; thus, it is difficult to obtain reflectance data with both high spatial resolution and frequent coverage from a single sensor. In this study, we propose a new Reflectance Bayesian Spatiotemporal Fusion Model (Ref-BSFM) using Landsat and MODIS (Moderate Resolution Imaging Spectroradiometer) surface reflectance, which is then used to construct reflectance datasets with high spatiotemporal resolution and a long time series. By comparing this model with other popular reconstruction methods (the Flexible Spatiotemporal Data Fusion Model, the Spatial and Temporal Adaptive Reflectance Fusion Model, and the Enhanced Spatial and Temporal Adaptive Reflectance Fusion Model), we demonstrate that our approach has the following advantages: (1) higher prediction accuracy, (2) effective treatment of cloud coverage, (3) insensitivity to the time span of data acquisition, (4) capture of temporal change information, and (5) higher retention of spatial details and inconspicuous MODIS patches. Reflectance time-series datasets generated by Ref-BSFM can be used to calculate a variety of remote-sensing-based vegetation indices, providing an important data source for land surface dynamic monitoring.

Remote Sensing Index for Mapping Canola Flowers Using MODIS Data

Mapping and tracing the changes in canola planting areas and yields in China are of great significance for macro-policy regulation and national food security. The bright yellow flower is a distinctive feature of canola, compared to other crops, and is also an important factor in predicting canola yield. Thus, yellowness indices were previously used to detect the canola flower using aerial imagery or median-resolution satellite data like Sentinel-2. However, it remains challenging to map the canola planting area and to trace long-term canola yields in China due to the wide areal extent of cultivation, different flowering periods in different locations and years, and the lack of high spatial resolution data within a long-term period. In this study, a novel canola index, called the enhanced area yellowness index (EAYI), for mapping canola flowers and based on Moderate Resolution Imaging Spectroradiometer (MODIS) time-series data, was developed. There are two improvements in the EAYI compared with previous studies. First, a method for estimating flowering period, based on geolocation and normalized difference vegetation index (NDVI) time-series, was established, to estimate the flowering period at each place in each year. Second, the EAYI enhances the weak flower signal in coarse pixels by combining the peak of yellowness index time-series and the valley of NDVI time-series during the estimated flowering period. With the proposed EAYI, canola flowering was mapped in five typical canola planting areas in China, during 2003-2017. Three different canola indices proposed previously, the normalized difference yellowness index (NDYI), ratio yellowness index (RYI) and Ashourloo canola index (Ashourloo CI), were also calculated for a comparison. Validation using the samples interpreted through higher resolution images demonstrated that the EAYI is better correlated with the reference canola coverage with R2 ranged from 0.31 to 0.70, compared to the previous indices with R2 ranged from 0.02 to 0.43. Compared with census canola yield data, the total EAYI was well correlated with actual yield in Jingmen, Yili and Hulun Buir, and well correlated with meteorological yields in all five study areas. In contrast, previous canola indices show a very low or even a negative correlation with both actual and meteorological yields. These results indicate that the EAYI is a potential index for mapping and tracing the change in canola areas, or yields, with MODIS data.

A Simple Spatio–Temporal Data Fusion Method Based on Linear Regression Coefficient Compensation

High spatio–temporal resolution remote sensing images are of great significance in the dynamic monitoring of the Earth’s surface. However, due to cloud contamination and the hardware limitations of sensors, it is difficult to obtain image sequences with both high spatial and temporal resolution. Combining coarse resolution images, such as the moderate resolution imaging spectroradiometer (MODIS), with fine spatial resolution images, such as Landsat or Sentinel-2, has become a popular means to solve this problem. In this paper, we propose a simple and efficient enhanced linear regression spatio–temporal fusion method (ELRFM), which uses fine spatial resolution images acquired at two reference dates to establish a linear regression model for each pixel and each band between the image reflectance and the acquisition date. The obtained regression coefficients are used to help allocate the residual error between the real coarse resolution image and the simulated coarse resolution image upscaled by the high spatial resolution result of the linear prediction. The developed method consists of four steps: (1) linear regression (LR), (2) residual calculation, (3) distribution of the residual and (4) singular value correction. The proposed method was tested in different areas and using different sensors. The results show that, compared to the spatial and temporal adaptive reflectance fusion model (STARFM) and the flexible spatio–temporal data fusion (FSDAF) method, the ELRFM performs better in capturing small feature changes at the fine image scale and has high prediction accuracy. For example, in the red band, the proposed method has the lowest root mean square error (RMSE) (ELRFM: 0.0123 vs. STARFM: 0.0217 vs. FSDAF: 0.0224 vs. LR: 0.0221). Furthermore, the lightweight algorithm design and calculations based on the Google Earth Engine make the proposed method computationally less expensive than the STARFM and FSDAF.

An Effective High Spatiotemporal Resolution NDVI Fusion Model Based on Histogram Clustering

The normalized difference vegetation index (NDVI) is a powerful tool for understanding past vegetation, monitoring the current state, and predicting its future. Due to technological and budget limitations, the existing global NDVI time-series data cannot simultaneously meet the needs of high spatial and temporal resolution. This study proposes a high spatiotemporal resolution NDVI fusion model based on histogram clustering (NDVI_FMHC), which uses a new spatiotemporal fusion framework to predict phenological and shape changes. Meanwhile, this model also uses four strategies to reduce error, including the construction of an overdetermined linear mixed model, multiscale prediction, residual distribution, and Gaussian filtering. Five groups of real MODIS_NDVI and Landsat_NDVI datasets were used to verify the predictive performance of the NDVI_FMHC. The results indicate that NDVI_FMHC has higher accuracy and robustness in forest areas (r = 0.9488 and ADD = 0.0229) and cultivated land areas (r = 0.9493 and ADD = 0.0605), while the prediction effect is relatively weak in areas subject to shape changes, such as flooded areas (r = 0.8450 and ADD = 0.0968), urban areas (r = 0.8855 and ADD = 0.0756), and fire areas (r = 0.8417 and ADD = 0.0749). Compared with ESTARFM, NDVI_LMGM, and FSDAF, NDVI_FMHC has the highest prediction accuracy, the best spatial detail retention, and the strongest ability to capture shape changes. Therefore, the NDVI_FMHC can obtain NDVI time-series data with high spatiotemporal resolution, which can be used to realize long-term land surface dynamic process research in a complex environment.

An Improved Spatiotemporal Data Fusion Method Using Surface Heterogeneity Information Based on ESTARFM

The use of the spatiotemporal data fusion method as an effective data interpolation method has received extensive attention in remote sensing (RS) academia. The enhanced spatial and temporal adaptive reflectance fusion model (ESTARFM) is one of the most famous spatiotemporal data fusion methods, as it is widely used to generate synthetic data. However, the ESTARFM algorithm uses moving windows with a fixed size to get the information around the central pixel, which hampers the efficiency and precision of spatiotemporal data fusion. In this paper, a modified ESTARFM data fusion algorithm that integrated the surface spatial information via a statistical method was developed. In the modified algorithm, the local variance of pixels around the central one was used as an index to adaptively determine the window size. Satellite images from two regions were acquired by employing the ESTARFM and modified algorithm. Results showed that the images predicted using the modified algorithm obtained more details than ESTARFM, as the frequency of pixels with the absolute difference of mean value of six bands’ reflectance between true observed image and predicted between 0 and 0.04 were 78% by ESTARFM and 85% by modified algorithm, respectively. In addition, the efficiency of the modified algorithm improved and the verification test showed the robustness of the modified algorithm. These promising results demonstrated the superiority of the modified algorithm to provide synthetic images compared with ESTARFM. Our research enriches the spatiotemporal data fusion method, and the automatic selection of moving window strategy lays the foundation of automatic processing of spatiotemporal data fusion on a large scale.

STAIR 2.0: A Generic and Automatic Algorithm to Fuse Modis, Landsat, and Sentinel-2 to Generate 10 m, Daily, and Cloud-/Gap-Free Surface Reflectance Product

Remote sensing datasets with both high spatial and high temporal resolution are critical for monitoring and modeling the dynamics of land surfaces. However, no current satellite sensor could simultaneously achieve both high spatial resolution and high revisiting frequency. Therefore, the integration of different sources of satellite data to produce a fusion product has become a popular solution to address this challenge. Many methods have been proposed to generate synthetic images with rich spatial details and high temporal frequency by combining two types of satellite datasets—usually frequent coarse-resolution images (e.g., MODIS) and sparse fine-resolution images (e.g., Landsat). In this paper, we introduce STAIR 2.0, a new fusion method that extends the previous STAIR fusion framework, to fuse three types of satellite datasets, including MODIS, Landsat, and Sentinel-2. In STAIR 2.0, input images are first processed to impute missing-value pixels that are due to clouds or sensor mechanical issues using a gap-filling algorithm. The multiple refined time series are then integrated stepwisely, from coarse- to fine- and high-resolution, ultimately providing a synthetic daily, high-resolution surface reflectance observations. We applied STAIR 2.0 to generate a 10-m, daily, cloud-/gap-free time series that covers the 2017 growing season of Saunders County, Nebraska. Moreover, the framework is generic and can be extended to integrate more types of satellite data sources, further improving the quality of the fusion product.

Deep Learning for Land Use and Land Cover Classification Based on Hyperspectral and Multispectral Earth Observation Data: A Review

Lately, with deep learning outpacing the other machine learning techniques in classifying images, we have witnessed a growing interest of the remote sensing community in employing these techniques for the land use and land cover classification based on multispectral and hyperspectral images; the number of related publications almost doubling each year since 2015 is an attest to that. The advances in remote sensing technologies, hence the fast-growing volume of timely data available at the global scale, offer new opportunities for a variety of applications. Deep learning being significantly successful in dealing with Big Data, seems to be a great candidate for exploiting the potentials of such complex massive data. However, there are some challenges related to the ground-truth, resolution, and the nature of data that strongly impact the performance of classification. In this paper, we review the use of deep learning in land use and land cover classification based on multispectral and hyperspectral images and we introduce the available data sources and datasets used by literature studies; we provide the readers with a framework to interpret the-state-of-the-art of deep learning in this context and offer a platform to approach methodologies, data, and challenges of the field.

Modelling Crop Biomass from Synthetic Remote Sensing Time Series: Example for the DEMMIN Test Site, Germany

This study compares the performance of the five widely used crop growth models (CGMs): World Food Studies (WOFOST), Coalition for Environmentally Responsible Economies (CERES)-Wheat, AquaCrop, cropping systems simulation model (CropSyst), and the semi-empiric light use efficiency approach (LUE) for the prediction of winter wheat biomass on the Durable Environmental Multidisciplinary Monitoring Information Network (DEMMIN) test site, Germany. The study focuses on the use of remote sensing (RS) data, acquired in 2015, in CGMs, as they offer spatial information on the actual conditions of the vegetation. Along with this, the study investigates the data fusion of Landsat (30 m) and Moderate Resolution Imaging Spectroradiometer (MODIS) (500 m) data using the spatial and temporal reflectance adaptive reflectance fusion model (STARFM) fusion algorithm. These synthetic RS data offer a 30-m spatial and one-day temporal resolution. The dataset therefore provides the necessary information to run CGMs and it is possible to examine the fine-scale spatial and temporal changes in crop phenology for specific fields, or sub sections of them, and to monitor crop growth daily, considering the impact of daily climate variability. The analysis includes a detailed comparison of the simulated and measured crop biomass. The modelled crop biomass using synthetic RS data is compared to the model outputs using the original MODIS time series as well. On comparison with the MODIS product, the study finds the performance of CGMs more reliable, precise, and significant with synthetic time series. Using synthetic RS data, the models AquaCrop and LUE, in contrast to other models, simulate the winter wheat biomass best, with an output of high R2 (>0.82), low RMSE (<600 g/m2) and significant p-value (<0.05) during the study period. However, inputting MODIS data makes the models underperform, with low R2 (<0.68) and high RMSE (>600 g/m2). The study shows that the models requiring fewer input parameters (AquaCrop and LUE) to simulate crop biomass are highly applicable and precise. At the same time, they are easier to implement than models, which need more input parameters (WOFOST and CERES-Wheat).

Fusion of MODIS and Landsat-Like Images for Daily High Spatial Resolution NDVI

One of the obstacles in monitoring agricultural crops is the difficulty in understanding and mapping rapid changes of these crops. With the purpose of addressing this issue, this study aimed to model and fuse the Moderate Resolution Imaging Spectroradiometer (MODIS) Normalized Difference Vegetation Index (NDVI) using Landsat-like images to achieve daily high spatial resolution NDVI. The study was performed for the period of 2017 on a commercial farm of irrigated maize-soybean rotation in the western region of the state of Bahia, Brazil. To achieve the objective, the following procedures were performed: (i) Landsat-like images were upscaled to match the Landsat-8 spatial resolution (30 m); (ii) the reflectance of Landsat-like images was intercalibrated using the Landsat-8 as a reference; (iii) Landsat-like reflectance images were upscaled to match the MODIS sensor spatial resolution (250 m); (iv) regression models were trained daily to model MODIS NDVI using the upscaled Landsat-like reflectance images (250 m) of the closest day as the input; and (v) the intercalibrated version of the Landsat-like images (30 m) used in the previous step was used as the input for the trained model, resulting in a downscaled MODIS NDVI (30 m). To determine the best fitting model, we used the following statistical metrics: coefficient of determination (r2), root mean square error (RMSE), Nash–Sutcliffe efficiency index (NSE), mean bias error (MBE), and mean absolute error (MAE). Among the assessed regression models, the Cubist algorithm was sensitive to changes in agriculture and performed best in modeling of the Landsat-like MODIS NDVI. The results obtained in the present research are promising and can enable the monitoring of dynamic phenomena with images available free of charge, changing the way in which decisions are made using satellite images.

A Learning-Enhanced Two-Pair Spatiotemporal Reflectance Fusion Model for GF-2 and GF-1 WFV Satellite Data

Since requirements of related applications for time series remotely-sensed images with high spatial resolution have been hard to be satisfied under current observation conditions of satellite sensors, it is key to reconstruct high-resolution images at specified dates. As an effective data reconstruction technique, spatiotemporal fusion can be used to generate time series land surface parameters with a clear geophysical significance. In this study, an improved fusion model based on the Sparse Representation-Based Spatiotemporal Reflectance Fusion Model (SPSTFM) is developed and assessed with reflectance data from Gaofen-2 Multi-Spectral (GF-2 MS) and Gaofen-1 Wide-Field-View (GF-1 WFV). By introducing a spatially enhanced training method to dictionary training and sparse coding processes, the developed fusion framework is expected to promote the description of high-resolution and low-resolution overcomplete dictionaries. Assessment indices including Average Absolute Deviation (AAD), Root-Mean-Square Error (RMSE), Peak Signal to Noise Ratio (PSNR), Correlation Coefficient (CC), spectral angle mapper (SAM), structure similarity (SSIM) and Erreur Relative Global Adimensionnelle de Synthèse (ERGAS) are then used to test employed fusion methods for a parallel comparison. The experimental results show that more accurate prediction of GF-2 MS reflectance than that from the SPSTFM can be obtained and furthermore comparable with popular two-pair based reflectance fusion models like the Spatial and Temporal Adaptive Fusion Model (STARFM) and the Enhanced-STARFM (ESTARFM).

A Novel Deep Learning-Based Spatiotemporal Fusion Method for Combining Satellite Images with Different Resolutions Using a Two-Stream Convolutional Neural Network

Spatiotemporal fusion is considered a feasible and cost-effective way to solve the trade-off between the spatial and temporal resolution of satellite sensors. Recently proposed learning-based spatiotemporal fusion methods can address the prediction of both phenological and land-cover change. In this paper, we propose a novel deep learning-based spatiotemporal data fusion method that uses a two-stream convolutional neural network. The method combines both forward and backward prediction to generate a target fine image, where temporal change-based and a spatial information-based mapping are simultaneously formed, addressing the prediction of both phenological and land-cover changes with better generalization ability and robustness. Comparative experimental results for the test datasets with phenological and land-cover changes verified the effectiveness of our method. Compared to existing learning-based spatiotemporal fusion methods, our method is more effective in predicting phenological change and directly reconstructing the prediction with complete spatial details without the need for auxiliary modulation.

Spatio-Temporal Sub-Pixel Land Cover Mapping of Remote Sensing Imagery Using Spatial Distribution Information From Same-Class Pixels

The generation of land cover maps with both fine spatial and temporal resolution would aid the monitoring of change on the Earth’s surface. Spatio-temporal sub-pixel land cover mapping (STSPM) uses a few fine spatial resolution (FR) maps and a time series of coarse spatial resolution (CR) remote sensing images as input to generate FR land cover maps with a temporal frequency of the CR data set. Traditional STSPM selects spatially adjacent FR pixels within a local window as neighborhoods to model the land cover spatial dependence, which can be a source of error and uncertainty in the maps generated by the analysis. This paper proposes a new STSPM using FR remote sensing images that pre- and/or post-date the CR image as ancillary data to enhance the quality of the FR map outputs. Spectrally similar pixels within the locality of a target FR pixel in the ancillary data are likely to represent the same land cover class and hence such same-class pixels can provide spatial information to aid the analysis. Experimental results showed that the proposed STSPM predicted land cover maps more accurately than two comparative state-of-the-art STSPM algorithms.