Deep Recurrent Neural Network for Agricultural Classification using multitemporal SAR Sentinel-1 for Camargue, France

Multi-Scale Semantic Segmentation and Spatial Relationship Recognition of Remote Sensing Images Based on an Attention Model

A comprehensive interpretation of remote sensing images involves not only remote sensing object recognition but also the recognition of spatial relations between objects. Especially in the case of different objects with the same spectrum, the spatial relationship can help interpret remote sensing objects more accurately. Compared with traditional remote sensing object recognition methods, deep learning has the advantages of high accuracy and strong generalizability regarding scene classification and semantic segmentation. However, it is difficult to simultaneously recognize remote sensing objects and their spatial relationship from end-to-end only relying on present deep learning networks. To address this problem, we propose a multi-scale remote sensing image interpretation network, called the MSRIN. The architecture of the MSRIN is a parallel deep neural network based on a fully convolutional network (FCN), a U-Net, and a long short-term memory network (LSTM). The MSRIN recognizes remote sensing objects and their spatial relationship through three processes. First, the MSRIN defines a multi-scale remote sensing image caption strategy and simultaneously segments the same image using the FCN and U-Net on different spatial scales so that a two-scale hierarchy is formed. The output of the FCN and U-Net are masked to obtain the location and boundaries of remote sensing objects. Second, using an attention-based LSTM, the remote sensing image captions include the remote sensing objects (nouns) and their spatial relationships described with natural language. Finally, we designed a remote sensing object recognition and correction mechanism to build the relationship between nouns in captions and object mask graphs using an attention weight matrix to transfer the spatial relationship from captions to objects mask graphs. In other words, the MSRIN simultaneously realizes the semantic segmentation of the remote sensing objects and their spatial relationship identification end-to-end. Experimental results demonstrated that the matching rate between samples and the mask graph increased by 67.37 percentage points, and the matching rate between nouns and the mask graph increased by 41.78 percentage points compared to before correction. The proposed MSRIN has achieved remarkable results.

A Novel Spatio-Temporal FCN-LSTM Network for Recognizing Various Crop Types Using Multi-Temporal Radar Images

In recent years, analyzing Synthetic Aperture Radar (SAR) data has turned into one of the challenging and interesting topics in remote sensing. Radar sensors are capable of imaging Earth’s surface independently of the weather conditions, local time of day, penetrating of waves through clouds, and containing spatial information on agricultural crop types. Based on these characteristics, the main goal sought in this research is to reveal the SAR imaging data capability in recognizing various agricultural crops in the main growth season in a more clarified and detailed way by using a deep-learning-based method. In the present research, the multi-temporal C-band Sentinel 1 images were used to classify 14 major classes of agricultural crops plus background in Denmark. By considering the capability of a deep learning method in analyzing satellite images, a novel, optimal, and lightweight network structure was developed and implemented based on a combination of a fully convolutional network (FCN) and a convolutional long short-term memory (ConvLSTM) network. The average pixel-based accuracy and Intersection over Union obtained from the proposed network were 86% and 0.64, respectively. Winter rapeseed, winter barley, winter wheat, spring barley, and sugar beet had the highest pixel-based accuracies of 95%, 94%, 93%, 90%, and 90%; respectively. The pixel-based accuracies for eight crop types and the background class were more than 84%. The network prediction showed that in field borders the classification confidence was lower than the center regions of the fields. However, the proposed structure has been able to identify different crops in multi-temporal Sentinel 1 data of a large area of around 254 thousand hectares with high performance.

Mapping Paddy Rice Using Sentinel-1 SAR Time Series in Camargue, France

This study proposes an effective method to map rice crops using the Sentinel-1 SAR (Synthetic Aperture Radar) time series over the Camargue region, Southern France. First, the temporal behavior of the SAR backscattering coefficient over 832 plots containing different crop types was analyzed. Through this analysis, the rice cultivation was identified using metrics derived from the Gaussian profile of the VV/VH time series (3 metrics), the variance of the VV/VH time series (one metric), and the slope of the linear regression of the VH time series (one metric). Using the derived metrics, rice plots were mapped through two different approaches: decision tree and Random Forest (RF). To validate the accuracy of each approach, the classified rice map was compared to the available national data. Similar high overall accuracy was obtained using both approaches. The overall accuracy obtained using a simple decision tree reached 96.3%, whereas an overall accuracy of 96.6% was obtained using the RF classifier. The approach, therefore, provides a simple yet precise and powerful tool to map paddy rice areas.

A New CNN-Bayesian Model for Extracting Improved Winter Wheat Spatial Distribution from GF-2 imagery

When the spatial distribution of winter wheat is extracted from high-resolution remote sensing imagery using convolutional neural networks (CNN), field edge results are usually rough, resulting in lowered overall accuracy. This study proposed a new per-pixel classification model using CNN and Bayesian models (CNN-Bayesian model) for improved extraction accuracy. In this model, a feature extractor generates a feature vector for each pixel, an encoder transforms the feature vector of each pixel into a category-code vector, and a two-level classifier uses the difference between elements of category-probability vectors as the confidence value to perform per-pixel classifications. The first level is used to determine the category of a pixel with high confidence, and the second level is an improved Bayesian model used to determine the category of low-confidence pixels. The CNN-Bayesian model was trained and tested on Gaofen 2 satellite images. Compared to existing models, our approach produced an improvement in overall accuracy, the overall accuracy of SegNet, DeepLab, VGG-Ex, and CNN-Bayesian was 0.791, 0.852, 0.892, and 0.946, respectively. Thus, this approach can produce superior results when winter wheat spatial distribution is extracted from satellite imagery.

Temporal Convolutional Neural Network for the Classification of Satellite Image Time Series

Latest remote sensing sensors are capable of acquiring high spatial and spectral Satellite Image Time Series (SITS) of the world. These image series are a key component of classification systems that aim at obtaining up-to-date and accurate land cover maps of the Earth’s surfaces. More specifically, current SITS combine high temporal, spectral and spatial resolutions, which makes it possible to closely monitor vegetation dynamics. Although traditional classification algorithms, such as Random Forest (RF), have been successfully applied to create land cover maps from SITS, these algorithms do not make the most of the temporal domain. This paper proposes a comprehensive study of Temporal Convolutional Neural Networks (TempCNNs), a deep learning approach which applies convolutions in the temporal dimension in order to automatically learn temporal (and spectral) features. The goal of this paper is to quantitatively and qualitatively evaluate the contribution of TempCNNs for SITS classification, as compared to RF and Recurrent Neural Networks (RNNs) —a standard deep learning approach that is particularly suited to temporal data. We carry out experiments on Formosat-2 scene with 46 images and one million labelled time series. The experimental results show that TempCNNs are more accurate than the current state of the art for SITS classification. We provide some general guidelines on the network architecture, common regularization mechanisms, and hyper-parameter values such as batch size; we also draw out some differences with standard results in computer vision (e.g., about pooling layers). Finally, we assess the visual quality of the land cover maps produced by TempCNNs.

Classification of Crops, Pastures, and Tree Plantations along the Season with Multi-Sensor Image Time Series in a Subtropical Agricultural Region

Timely and efficient land-cover mapping is of high interest, especially in agricultural landscapes. Classification based on satellite images over the season, while important for cropland monitoring, remains challenging in subtropical agricultural areas due to the high diversity of management systems and seasonal cloud cover variations. This work presents supervised object-based classifications over the year at 2-month time-steps in a heterogeneous region of 12,000 km2 in the Sao Paulo region of Brazil. Different methods and remote-sensing datasets were tested with the random forest algorithm, including optical and radar data, time series of images, and cloud gap-filling methods. The final selected method demonstrated an overall accuracy of approximately 0.84, which was stable throughout the year, at the more detailed level of classification; confusion mainly occurred among annual crop classes and soil classes. We showed in this study that the use of time series was useful in this context, mainly by including a small number of highly discriminant images. Such important images were eventually distant in time from the prediction date, and they corresponded to a high-quality image with low cloud cover. Consequently, the final classification accuracy was not sensitive to the cloud gap-filling method, and simple median gap-filling or linear interpolations with time were sufficient. Sentinel-1 images did not improve the classification results in this context. For within-season dynamic classes, such as annual crops, which were more difficult to classify, field measurement efforts should be densified and planned during the most discriminant window, which may not occur during the crop vegetation peak.

Multi-Temporal SAR Data Large-Scale Crop Mapping Based on U-Net Model

Due to the unique advantages of microwave detection, such as its low restriction from the atmosphere and its capability to obtain structural information about ground targets, synthetic aperture radar (SAR) is increasingly used in agricultural observations. However, while SAR data has shown great potential for large-scale crop mapping, there have been few studies on the use of SAR images for large-scale multispecies crop classification at present. In this paper, a large-scale crop mapping method using multi-temporal dual-polarization SAR data was proposed. To reduce multi-temporal SAR data redundancy, a multi-temporal images optimization method based on analysis of variance (ANOVA) and Jeffries–Matusita (J–M) distance was applied to the time series of images after preprocessing to select the optimal images. Facing the challenges from smallholder farming modes, which caused the complex crop planting patterns in the study area, U-Net, an improved fully convolutional network (FCN), was used to predict the different crop types. In addition, the batch normalization (BN) algorithm was introduced to the U-Net model to solve the problem of a large number of crops and unbalanced sample numbers, which had greatly improved the efficiency of network training. Finally, we conducted experiments using multi-temporal Sentinel-1 data from Fuyu City, Jilin Province, China in 2017, and we obtained crop mapping results with an overall accuracy of 85% as well as a Kappa coefficient of 0.82. Compared with the traditional machine learning methods (e.g., random forest (RF) and support vector machine (SVM)), the proposed method can still achieve better classification performance under the condition of a complex crop planting structure.

Ultra-Light Aircraft-Based Hyperspectral and Colour-Infrared Imaging to Identify Deciduous Tree Species in an Urban Environment

One may consider the application of remote sensing as a trade-off between the imaging platforms, sensors, and data gathering and processing techniques. This study addresses the potential of hyperspectral imaging using ultra-light aircraft for vegetation species mapping in an urban environment, exploring both the engineering and scientific aspects related to imaging platform design and image classification methods. An imaging system based on simultaneous use of Rikola frame format hyperspectral and Nikon D800E adopted colour infrared cameras installed onboard a Bekas X32 manned ultra-light aircraft is introduced. Two test imaging flight missions were conducted in July of 2015 and September of 2016 over a 4000 ha area in Kaunas City, Lithuania. Sixteen and 64 spectral bands in 2015 and 2016, respectively, in a spectral range of 500–900 nm were recorded with colour infrared images. Three research questions were explored assessing the identification of six deciduous tree species: (1) Pre-treatment of spectral features for classification, (2) testing five conventional machine learning classifiers, and (3) fusion of hyperspectral and colour infrared images. Classification performance was assessed by applying leave-one-out cross-validation at the individual crown level and using as a reference at least 100 field inventoried trees for each species. The best-performing classification algorithm—multilayer perceptron, using all spectral properties extracted from the hyperspectral images—resulted in a moderate classification accuracy. The overall classification accuracy was 63%, Cohen’s Kappa was 0.54, and the species-specific classification accuracies were in the range of 51–72%. Hyperspectral images resulted in significantly better tree species classification ability than the colour infrared images and simultaneous use of spectral properties extracted from hyperspectral and colour infrared images improved slightly the accuracy over the 2015 image. Even though classifications using hyperspectral data cubes of 64 bands resulted in relatively larger accuracies than with 16 bands, classification error matrices were not statistically different. Alternative imaging platforms (like an unmanned aerial vehicle and a Cessna 172 aircraft) and settings of the flights were discussed using simulated imaging projects assuming the same study area and field of application. Ultra-light aircraft-based hyperspectral and colour-infrared imaging was considered to be a technically and economically sound solution for urban green space inventories to facilitate tree mapping, characterization, and monitoring.

Synergistic Use of Radar Sentinel-1 and Optical Sentinel-2 Imagery for Crop Mapping: A Case Study for Belgium

A timely inventory of agricultural areas and crop types is an essential requirement for ensuring global food security and allowing early crop monitoring practices. Satellite remote sensing has proven to be an increasingly more reliable tool to identify crop types. With the Copernicus program and its Sentinel satellites, a growing source of satellite remote sensing data is publicly available at no charge. Here, we used joint Sentinel-1 radar and Sentinel-2 optical imagery to create a crop map for Belgium. To ensure homogenous radar and optical inputs across the country, Sentinel-1 12-day backscatter mosaics were created after incidence angle normalization, and Sentinel-2 normalized difference vegetation index (NDVI) images were smoothed to yield 10-daily cloud-free mosaics. An optimized random forest classifier predicted the eight crop types with a maximum accuracy of 82% and a kappa coefficient of 0.77. We found that a combination of radar and optical imagery always outperformed a classification based on single-sensor inputs, and that classification performance increased throughout the season until July, when differences between crop types were largest. Furthermore, we showed that the concept of classification confidence derived from the random forest classifier provided insight into the reliability of the predicted class for each pixel, clearly showing that parcel borders have a lower classification confidence. We concluded that the synergistic use of radar and optical data for crop classification led to richer information increasing classification accuracies compared to optical-only classification. Further work should focus on object-level classification and crop monitoring to exploit the rich potential of combined radar and optical observations.

Estimation of Rice Height and Biomass Using Multitemporal SAR Sentinel-1 for Camargue, Southern France

The research and improvement of methods to be used for crop monitoring are currently major challenges, especially for radar images due to their speckle noise nature. The European Space Agency’s (ESA) Sentinel-1 constellation provides synthetic aperture radar (SAR) images coverage with a 6-day revisit period at a high spatial resolution of pixel spacing of 20 m. Sentinel-1 data are considerably useful, as they provide valuable information of the vegetation cover. The objective of this work is to study the capabilities of multitemporal radar images for rice height and dry biomass retrievals using Sentinel-1 data. To do this, we train Sentinel-1 data against ground measurements with classical machine learning techniques (Multiple Linear Regression (MLR), Support Vector Regression (SVR) and Random Forest (RF)) to estimate rice height and dry biomass. The study is carried out on a multitemporal Sentinel-1 dataset acquired from May 2017 to September 2017 over the Camargue region, southern France. The ground in-situ measurements were made in the same period to collect rice height and dry biomass over 11 rice fields. The images were processed in order to produce a radar stack in C-band including dual-polarization VV (Vertical receive and Vertical transmit) and VH (Vertical receive and Horizontal transmit) data. We found that non-parametric methods (SVR and RF) had a better performance over the parametric MLR method for rice biophysical parameter retrievals. The accuracy of rice height estimation showed that rice height retrieval was strongly correlated to the in-situ rice height from dual-polarization, in which Random Forest yielded the best performance with correlation coefficient R 2 = 0.92 and the root mean square error (RMSE) 16% (7.9 cm). In addition, we demonstrated that the correlation of Sentinel-1 signal to the biomass was also very high in VH polarization with R 2 = 0.9 and RMSE = 18% (162 g·m − 2 ) (with Random Forest method). Such results indicate that the highly qualified Sentinel-1 radar data could be well exploited for rice biomass and height retrieval and they could be used for operational tasks.