Convolutional Neural Networks enable efficient, accurate and fine-grained segmentation of plant species and communities from high-resolution UAV imagery

Deep learning-based detection of indicator species for monitoring biodiversity in semi-natural grasslands

Deep learning (DL) has huge potential to provide valuable insights into biodiversity changes in species-rich agricultural ecosystems such as semi-natural grasslands, helping to prioritize and plan conservation efforts. However, DL has been underexplored in grassland conservation efforts, hindered by data scarcity, intricate ecosystem interactions, and limited economic incentives. Here, we developed a DL-based object-detection model to identify indicator species, a group of vascular plant species that serve as surrogates for biodiversity assessment in high nature value (HNV) grasslands. We selected indicator species Armeria maritima, Campanula patula, Cirsium oleraceum, and Daucus carota. To overcome the hurdle of limited data, we grew indicator plants under controlled greenhouse conditions, generating a sufficient dataset for DL model training. The model was initially trained on this greenhouse dataset. Then, smaller datasets derived from an experimental grassland plot and natural grasslands were added to the training to facilitate the transition from greenhouse to field conditions. Our optimized model achieved remarkable average precision (AP) on test datasets, with 98.6 AP50 on greenhouse data, 98.2 AP50 on experimental grassland data, and 96.5 AP50 on semi-natural grassland data. Our findings highlight the innovative application of greenhouse-grown specimens for the in-situ identification of plants, bolstering biodiversity monitoring in grassland ecosystems. Furthermore, the study illuminates the promising role of DL techniques in conservation programs, particularly as a monitoring tool to support result-based agri-environment schemes.

Automatic Scoring of Rhizoctonia Crown and Root Rot Affected Sugar Beet Fields from Orthorectified UAV Images Using Machine Learning

Rhizoctonia crown and root rot (RCRR), caused by Rhizoctonia solani, can cause severe yield and quality losses in sugar beet. The most common strategy to control the disease is the development of resistant varieties. In the breeding process, field experiments with artificial inoculation are carried out to evaluate the performance of genotypes and varieties. The phenotyping process in breeding trials requires constant monitoring and scoring by skilled experts. This work is time demanding and shows bias and heterogeneity according to the experience and capacity of each individual person. Optical sensors and artificial intelligence have demonstrated great potential to achieve higher accuracy than human raters and the possibility to standardize phenotyping applications. A workflow combining red-green-blue and multispectral imagery coupled to an unmanned aerial vehicle (UAV), as well as machine learning techniques, was applied to score diseased plants and plots affected by RCRR. Georeferenced annotation of UAV-orthorectified images was carried out. With the annotated images, five convolutional neural networks were trained to score individual plants. The training was carried out with different image analysis strategies and data augmentation. The custom convolutional neural network trained from scratch together with pretrained MobileNet showed the best precision in scoring RCRR (0.73 to 0.85). The average per plot of spectral information was used to score the plots, and the benefit of adding the information obtained from the score of individual plants was compared. For this purpose, machine learning models were trained together with data management strategies, and the best-performing model was chosen. A combined pipeline of random forest and k-nearest neighbors has shown the best weighted precision (0.67). This research provides a reliable workflow for detecting and scoring RCRR based on aerial imagery. RCRR is often distributed heterogeneously in trial plots; therefore, considering the information from individual plants of the plots showed a significant improvement in UAV-based automated monitoring routines.

Verification study on how macrofungal fruitbody formation can be predicted by artificial neural network

The occurrence and regularity of macrofungal fruitbody formation are influenced by meteorological conditions; however, there is a scarcity of data about the use of machine-learning techniques to estimate their occurrence based on meteorological indicators. Therefore, we employed an artificial neural network (ANN) to forecast fruitbody occurrence in mycorrhizal species of Russula and Amanita, utilizing meteorological factors and validating the accuracy of the forecast of fruitbody formation. Fungal data were collected from two locations in Western Hungary between 2015 and 2020. The ANN was the commonly used algorithm for classification problems: feed-forward multilayer perceptrons with a backpropagation algorithm to estimate the binary (Yes/No) classification of fruitbody appearance in natural and undisturbed forests. The verification indices resulted in two outcomes: however, development is most often studied by genus level, we established a more successful, new model per species. Furthermore, the algorithm is able to successfully estimate fruitbody formations with medium to high accuracy (60-80%). Therefore, this work was the first to reliably utilise the ANN approach of estimating fruitbody occurrence based on meteorological parameters of mycorrhizal specified with an extended vegetation period. These findings can assist in field mycological investigations that utilize sporocarp occurrences to ascertain species abundance.

Integrating carbon sequestration and biodiversity impacts in forested ecosystems: Concepts, cases, and policies

The challenges posed by climate change, biodiversity loss, and land-use are deeply interconnected and integrated solutions are needed. This paper presents results from 11 contributions to a special issue covering topics of integrated modeling and spatial prioritization, mass-balance studies, Earth Observation techniques, research infrastructure developments, and evaluation of policy measures and economic compensation schemes. The spatial scale of the studies ranges from detailed site-specific to a European scale. This paper briefly summarizes the main findings of these studies, makes some general overall conclusions, and identifies topics for further research and methods developments.

Consumer-grade UAV imagery facilitates semantic segmentation of species-rich savanna tree layers

Conventional forest inventories are labour-intensive. This limits the spatial extent and temporal frequency at which woody vegetation is usually monitored. Remote sensing provides cost-effective solutions that enable extensive spatial coverage and high sampling frequency. Recent studies indicate that convolutional neural networks (CNNs) can classify woody forests, plantations, and urban vegetation at the species level using consumer-grade unmanned aerial vehicle (UAV) imagery. However, whether such an approach is feasible in species-rich savanna ecosystems remains unclear. Here, we tested whether small data sets of high-resolution RGB orthomosaics suffice to train U-Net, FC-DenseNet, and DeepLabv3 + in semantic segmentation of savanna tree species. We trained these models on an 18-ha training area and explored whether models could be transferred across space and time. These models could recognise trees in adjacent (mean F1-Score = 0.68) and distant areas (mean F1-Score = 0.61) alike. Over time, a change in plant morphology resulted in a decrease of model accuracy. Our results show that CNN-based tree mapping using consumer-grade UAV imagery is possible in savanna ecosystems. Still, larger and more heterogeneous data sets can further improve model robustness to capture variation in plant morphology across time and space.

Image recognition of traditional Chinese medicine based on deep learning

Chinese herbal medicine is an essential part of traditional Chinese medicine and herbalism, and has important significance in the treatment combined with modern medicine. The correct use of Chinese herbal medicine, including identification and classification, is crucial to the life safety of patients. Recently, deep learning has achieved advanced performance in image classification, and researchers have applied this technology to carry out classification work on traditional Chinese medicine and its products. Therefore, this paper uses the improved ConvNeXt network to extract features and classify traditional Chinese medicine. Its structure is to fuse ConvNeXt with ACMix network to improve the performance of ConvNeXt feature extraction. Through using data processing and data augmentation techniques, the sample size is indirectly expanded, the generalization ability is enhanced, and the feature extraction ability is improved. A traditional Chinese medicine classification model is established, and the good recognition results are achieved. Finally, the effectiveness of traditional Chinese medicine identification is verified through the established classification model, and different depth of network models are compared to improve the efficiency and accuracy of the model.

Mapping Malaria Vector Habitats in West Africa: Drone Imagery and Deep Learning Analysis for Targeted Vector Surveillance

Disease control programs are needed to identify the breeding sites of mosquitoes, which transmit malaria and other diseases, in order to target interventions and identify environmental risk factors. The increasing availability of very-high-resolution drone data provides new opportunities to find and characterize these vector breeding sites. Within this study, drone images from two malaria-endemic regions in Burkina Faso and Côte d'Ivoire were assembled and labeled using open-source tools. We developed and applied a workflow using region-of-interest-based and deep learning methods to identify land cover types associated with vector breeding sites from very-high-resolution natural color imagery. Analysis methods were assessed using cross-validation and achieved maximum Dice coefficients of 0.68 and 0.75 for vegetated and non-vegetated water bodies, respectively. This classifier consistently identified the presence of other land cover types associated with the breeding sites, obtaining Dice coefficients of 0.88 for tillage and crops, 0.87 for buildings and 0.71 for roads. This study establishes a framework for developing deep learning approaches to identify vector breeding sites and highlights the need to evaluate how results will be used by control programs.

The spatial distribution and expansion of Eucalyptus in its hotspots: Implications on agricultural landscapes

Fast coppicing plantations like Eucalyptus are becoming an ever increasingly important land use system globally, including the Eucalyptus hotspot highlands of Northwestern Ethiopia. However, comprehensive information regarding species composition is essential for proper planning and policy decisions. The current study mapped the spatial distribution of Eucalyptus globulus (hereafter referred to as Eucalyptus) and identified the key push factors for its expansion. The study used a mapping procedure that uses Landsat imagery together with ground truth data based on supervised training of a pixel-by-pixel classification algorithm within image regions to distinguish areas of Eucalyptus plantations from other classes. High-resolution multispectral and multi-temporal remote-sensing images were combined with ground truth data to produce robust features of Eucalyptus plantation distribution maps. Heckman's Two-Stage econometric model was also employed for determining the major driving factors of Eucalyptus expansion. The results of the mapping algorithm were Eucalyptus plantation distribution maps of 30 × 30 m resolution that showed temporal changes from 1999 to 2021. The findings revealed that Eucalyptus coverage increased by 55% during the period from 1999 to 2010 and the change expressively increased to 69% in 2021 with respect to the reference period. The study also found that a number of push factors influenced the size of land planted with Eucalyptus. The developed maps showing the spatial distribution and expansion of Eucalyptus will help policymakers properly manage the ecosystems and agricultural landscapes of Eucalyptus growing areas.

Self-supervised learning for remote sensing scene classification under the few shot scenario

Scene classification is a crucial research problem in remote sensing (RS) that has attracted many researchers recently. It has many challenges due to multiple issues, such as: the complexity of remote sensing scenes, the classes overlapping (as a scene may contain objects that belong to foreign classes), and the difficulty of gaining sufficient labeled scenes. Deep learning (DL) solutions and in particular convolutional neural networks (CNN) are now state-of-the-art solution in RS scene classification; however, CNN models need huge amounts of annotated data, which can be costly and time-consuming. On the other hand, it is relatively easy to acquire large amounts of unlabeled images. Recently, Self-Supervised Learning (SSL) is proposed as a method that can learn from unlabeled images, potentially reducing the need for labeling. In this work, we propose a deep SSL method, called RS-FewShotSSL, for RS scene classification under the few shot scenario when we only have a few (less than 20) labeled scenes per class. Under this scenario, typical DL solutions that fine-tune CNN models, pre-trained on the ImageNet dataset, fail dramatically. In the SSL paradigm, a DL model is pre-trained from scratch during the pretext task using the large amounts of unlabeled scenes. Then, during the main or the so-called downstream task, the model is fine-tuned on the labeled scenes. Our proposed RS-FewShotSSL solution is composed of an online network and a target network both using the EfficientNet-B3 CNN model as a feature encoder backbone. During the pretext task, RS-FewShotSSL learns discriminative features from the unlabeled images using cross-view contrastive learning. Different views are generated from each image using geometric transformations and passed to the online and target networks. Then, the whole model is optimized by minimizing the cross-view distance between the online and target networks. To address the problem of limited computation resources available to us, our proposed method uses a novel DL architecture that can be trained using both high-resolution and low-resolution images. During the pretext task, RS-FewShotSSL is trained using low-resolution images, thereby, allowing for larger batch sizes which significantly boosts the performance of the proposed pipeline on the task of RS classification. In the downstream task, the target network is discarded, and the online network is fine-tuned using the few labeled shots or scenes. Here, we use smaller batches of both high-resolution and low-resolution images. This architecture allows RS-FewshotSSL to benefit from both large batch sizes and full image sizes, thereby learning from the large amounts of unlabeled data in an effective way. We tested RS-FewShotSSL on three RS public datasets, and it demonstrated a significant improvement compared to other state-of-the-art methods such as: SimCLR, MoCo, BYOL and IDSSL.

The use of drones for mosquito surveillance and control

In recent years, global health security has been threatened by the geographical expansion of vector-borne infectious diseases such as malaria, dengue, yellow fever, Zika and chikungunya. For a range of these vector-borne diseases, an increase in residual (exophagic) transmission together with ecological heterogeneity in everything from weather to local human migration and housing to mosquito species' behaviours presents many challenges to effective mosquito control. The novel use of drones (or uncrewed aerial vehicles) may play a major role in the success of mosquito surveillance and control programmes in the coming decades since the global landscape of mosquito-borne diseases and disease dynamics fluctuates frequently and there could be serious public health consequences if the issues of insecticide resistance and outdoor transmission are not adequately addressed. For controlling both aquatic and adult stages, for several years now remote sensing data have been used together with predictive modelling for risk, incidence and detection of transmission hot spots and landscape profiles in relation to mosquito-borne pathogens. The field of drone-based remote sensing is under continuous change due to new technology development, operation regulations and innovative applications. In this review we outline the opportunities and challenges for integrating drones into vector surveillance (i.e. identification of breeding sites or mapping micro-environmental composition) and control strategies (i.e. applying larval source management activities or deploying genetically modified agents) across the mosquito life-cycle. We present a five-step systematic environmental mapping strategy that we recommend be undertaken in locations where a drone is expected to be used, outline the key considerations for incorporating drone or other Earth Observation data into vector surveillance and provide two case studies of the advantages of using drones equipped with multispectral cameras. In conclusion, recent developments mean that drones can be effective for accurately conducting surveillance, assessing habitat suitability for larval and/or adult mosquitoes and implementing interventions. In addition, we briefly discuss the need to consider permissions, costs, safety/privacy perceptions and community acceptance for deploying drone activities.

Automated mapping of Portulacaria afra canopies for restoration monitoring with convolutional neural networks and heterogeneous unmanned aerial vehicle imagery

Ecosystem restoration and reforestation often operate at large scales, whereas monitoring practices are usually limited to spatially restricted field measurements that are (i) time- and labour-intensive, and (ii) unable to accurately quantify restoration success over hundreds to thousands of hectares. Recent advances in remote sensing technologies paired with deep learning algorithms provide an unprecedented opportunity for monitoring changes in vegetation cover at spatial and temporal scales. Such data can feed directly into adaptive management practices and provide insights into restoration and regeneration dynamics. Here, we demonstrate that convolutional neural network (CNN) segmentation algorithms can accurately classify the canopy cover of Portulacaria afra Jacq. in imagery acquired using different models of unoccupied aerial vehicles (UAVs) and under variable light intensities. Portulacaria afra is the target species for the restoration of Albany Subtropical Thicket vegetation, endemic to South Africa, where canopy cover is challenging to measure due to the dense, tangled structure of this vegetation. The automated classification strategy presented here is widely transferable to restoration monitoring as its application does not require any knowledge of the CNN model or specialist training, and can be applied to imagery generated by a range of UAV models. This will reduce the sampling effort required to track restoration trajectories in space and time, contributing to more effective management of restoration sites, and promoting collaboration between scientists, practitioners and landowners.

Artificial neural network-based model to predict the effect of γ-aminobutyric acid on salinity and drought responsive morphological traits in pomegranate

Recently, γ-Aminobutyric acid (GABA) has been introduced as a treatment with high physiological activity induction to enhance the ability of plants against drought and salinity stress, which led to a decline in plant growth. Since changes in morphological traits to drought and salinity stress are influenced by multiple factors, advanced computational analysis has great potential for computing nonlinear and multivariate data. In this work, the effect of four input variables including GABA concentration, pomegranate cultivars, days of treatment, and drought and salinity stress evaluated to predict and modeling of morphological traits using artificial neural network (ANN) models including multilayer perceptron (MLP) and radial basis function (RBF). Image processing technique was used to measure the LLI, LWI, and LAI parameters. Among the ANNs applied, the MLP algorithm was chosen as the best model based on the highest accuracy. Furthermore, to predict and estimate the optimal values of input variables for achieving the best morphological parameters, the MLP algorithm was linked to a non-dominated sorting genetic algorithm-II (NSGA-II). Based on the results of MLP-NSGA-II, the best values of crown diameter (18.42 cm), plant height (151.82 cm), leaf length index (5.67 cm), leaf width index (1.76 cm), and leaf area index (13.82 cm) could be achieved with applying 10.57 mM GABA on ‘Atabaki’ cultivar under control (non-stress) condition after 20.8 days. The results of modeling and optimization can be helpful to predict the morphological responses to drought and salinity conditions.

Individual Tree Crown Segmentation and Crown Width Extraction From a Heightmap Derived From Aerial Laser Scanning Data Using a Deep Learning Framework

Deriving individual tree crown (ITC) information from light detection and ranging (LiDAR) data is of great significance to forest resource assessment and smart management. After proof-of-concept studies, advanced deep learning methods have been shown to have high efficiency and accuracy in remote sensing data analysis and geoscience problem solving. This study proposes a novel concept for synergetic use of the YOLO-v4 deep learning network based on heightmaps directly generated from airborne LiDAR data for ITC segmentation and a computer graphics algorithm for refinement of the segmentation results involving overlapping tree crowns. This concept overcomes the limitations experienced by existing ITC segmentation methods that use aerial photographs to obtain texture and crown appearance information and commonly encounter interference due to heterogeneous solar illumination intensities or interlacing branches and leaves. Three generative adversarial networks (WGAN, CycleGAN, and SinGAN) were employed to generate synthetic images. These images were coupled with manually labeled training samples to train the network. Three forest plots, namely, a tree nursery, forest landscape and mixed tree plantation, were used to verify the effectiveness of our approach. The results showed that the overall recall of our method for detecting ITCs in the three forest plot types reached 83.6%, with an overall precision of 81.4%. Compared with reference field measurement data, the coefficient of determination R 2 was ≥ 79.93% for tree crown width estimation, and the accuracy of our deep learning method was not influenced by the values of key parameters, yielding 3.9% greater accuracy than the traditional watershed method. The results demonstrate an enhancement of tree crown segmentation in the form of a heightmap for different forest plot types using the concept of deep learning, and our method bypasses the visual complications arising from aerial images featuring diverse textures and unordered scanned points with irregular geometrical properties.

Cloud-Based Monitoring and Evaluation of the Spatial-Temporal Distribution of Southeast Asia’s Mangroves Using Deep Learning

This paper proposes a cloud-based mangrove monitoring framework that uses Google Collaboratory and Google Earth Engine to classify mangroves in Southeast Asia (SEA) using satellite remote sensing imagery (SRSI). Three multi-class classification convolutional neural network (CNN) models were generated, showing F1-score values as high as 0.9 in only six epochs of training. Mangrove forests are tropical and subtropical environments that provide essential ecosystem services to local biota and coastal communities and are considered the most efficient vegetative carbon stock globally. Despite their importance, mangrove forest cover continues to decline worldwide, especially in SEA. Scientists have produced monitoring tools based on SRSI and CNNs to identify deforestation hotspots and drive targeted interventions. Nevertheless, although CNNs excel in distinguishing between different landcover types, their greatest limitation remains the need for significant computing power to operate. This may not always be feasible, especially in developing countries. The proposed framework is believed to provide a robust, low-cost, cloud-based, near-real-time monitoring tool that could serve governments, environmental agencies, and researchers, to help map mangroves in SEA.

The Role of Environmental Water and Reedbed Condition on the Response of Phragmites australis Reedbeds to Flooding

Globally, wetlands have experienced significant declines in area and condition. Reedbeds are a key attribute of many wetlands and are typically composed of Phragmites australis (common reed), a globally distributed emergent aquatic perennial grass. Environmental water is increasingly used to support functioning river and floodplain ecosystems, including reedbeds, where maintaining wetland vegetation condition is a common objective. Drone-based remote sensing allows for the consistent collection of high-quality data in locations such as wetlands where access is limited. We used unoccupied aerial vehicles (UAVs) and convolutional neural networks (CNNs) to estimate the cover of Phragmites australis and examine the role of reedbed condition and prior environmental watering in the response of reedbeds to flooding. Data were collected from a large inland reedbed in semi-arid western New South Wales, Australia between October 2019 and March 2021 using UAVs and processed using CNNs. Prior to the flood event, sites that had received environmental water had a significantly greater cover of Phragmites australis. The sites that were not managed with environmental water had very low cover (<1%) of reeds prior to the flood event and transitioned from a Critical condition to a Poor or Medium condition following flooding. Using UAVs and CNNs we demonstrated the role environmental water plays in filling the gaps between large flood events and maintaining the condition and resilience of reedbeds.

How Does Sample Labeling and Distribution Affect the Accuracy and Efficiency of a Deep Learning Model for Individual Tree-Crown Detection and Delineation

Monitoring and assessing vegetation using deep learning approaches has shown promise in forestry applications. Sample labeling to represent forest complexity is the main limitation for deep learning approaches for remote sensing vegetation classification applications, and few studies have focused on the impact of sample labeling methods on model performance and model training efficiency. This study is the first-of-its-kind that uses Mask region-based convolutional neural networks (Mask R-CNN) to evaluate the influence of sample labeling methods (including sample size and sample distribution) on individual tree-crown detection and delineation. A flight was conducted over a plantation with Fokienia hodginsii as the main tree species using a Phantom4-Multispectral (P4M) to obtain UAV imagery, and a total of 2061 manually and accurately delineated tree crowns were used for training and validating (1689) and testing (372). First, the model performance of three pre-trained backbones (ResNet-34, ResNet-50, and ResNet-101) was evaluated. Second, random deleting and clumped deleting methods were used to repeatedly delete 10% from the original sample set to reduce the training and validation set, to simulate two different sample distributions (the random sample set and the clumped sample set). Both RGB image and Multi-band images derived from UAV flights were used to evaluate model performance. Each model’s average per-epoch training time was calculated to evaluate the model training efficiency. The results showed that ResNet-50 yielded a more robust network than ResNet-34 and ResNet-101 when the same parameters were used for Mask R-CNN. The sample size determined the influence of sample labeling methods on the model performance. Random sample labeling had lower requirements for sample size compared to clumped sample labeling, and unlabeled trees in random sample labeling had no impact on model training. Additionally, the model with clumped samples provides a shorter average per-epoch training time than the model with random samples. This study demonstrates that random sample labeling can greatly reduce the requirement of sample size, and it is not necessary to accurately label each sample in the image during the sample labeling process.

Airborne HySpex Hyperspectral Versus Multitemporal Sentinel-2 Images for Mountain Plant Communities Mapping

Climate change and anthropopression significantly impact plant communities by leading to the spread of expansive and alien invasive plants, thus reducing their biodiversity. Due to significant elevation gradients, high-mountain plant communities in a small area allow for the monitoring of the most important environmental changes. Additionally, being a tourist attraction, they are exposed to direct human influence (e.g., trampling). Airborne hyperspectral remote sensing is one of the best data sources for vegetation mapping, but flight campaign costs limit the repeatability of surveys. A possible alternative approach is to use satellite data from the Copernicus Earth observation program. In our study, we compared multitemporal Sentinel-2 data with HySpex airborne hyperspectral images to map the plant communities on Tatra Mountains based on open-source R programing implementation of Random Forest and Support Vector Machine classifiers. As high-mountain ecosystems are adapted to topographic conditions, the input of Digital Elevation Model (DEM) derivatives on the classification accuracy was analyzed and the effect of the number of training pixels was tested to procure practical information for field campaign planning. For 13 classes (from rock scree communities and alpine grasslands to montane conifer and deciduous forests), we achieved results in the range of 76–90% F1-score depending on the data set. Topographic features: digital terrain model (DTM), normalized digital surface model (nDSM), and aspect and slope maps improved the accuracy of HySpex spectral images, transforming their minimum noise fraction (MNF) bands and Sentinel-2 data sets by 5–15% of the F1-score. Maps obtained on the basis of HySpex imagery (2 m; 430 bands) had a high similarity to maps obtained on the basis of multitemporal Sentinel-2 data (10 m; 132 bands; 11 acquisition dates), which was less than one percentage point for classifications based on 500–1000 pixels; for sets consisting of 50–100 pixels, Random Forest (RF) offered better accuracy.

Machine Learning Approaches in Traditional Chinese Medicine: A Systematic Review

Machine learning (ML), as a branch of artificial intelligence, acquires the potential and meaningful rules from the mass of data via diverse algorithms. Owing to all research of traditional Chinese medicine (TCM) belonging to the digitalization of clinical records or experimental works, a massive and complex amount of data has become an inextricable part of the related studies. It is thus not surprising that ML approaches, as novel and efficient tools to mine the useful knowledge from data, have created inroads in a diversity of scopes of TCM over the past decade of years. However, by browsing lots of literature, we find that not all of the ML approaches perform well in the same field. Upon further consideration, we infer that the specificity may inhere between the ML approaches and their applied fields. This systematic review focuses its attention on the four categories of ML approaches and their eight application scopes in TCM. According to the function, ML approaches are classified into four categories, including classification, regression, clustering, and dimensionality reduction, and into 14 models as follows in more detail: support vector machine, least square-support vector machine, logistic regression, partial least squares regression, k-means clustering, hierarchical cluster analysis, artificial neural network, back propagation neural network, convolutional neural network, decision tree, random forest, principal component analysis, partial least squares-discriminant analysis, and orthogonal partial least squares-discriminant analysis. The eight common applied fields are divided into two parts: one for TCM, such as the diagnosis of diseases, the determination of syndromes, and the analysis of prescription, and the other for the related researches of Chinese herbal medicine, such as the quality control, the identification of geographic origins, the pharmacodynamic material basis, the medicinal properties, and the pharmacokinetics and pharmacodynamics. Additionally, this paper discusses the function and feature difference among ML approaches when they are applied to the corresponding fields via comparing their principles. The specificity of each approach to its applied fields has also been affirmed, whereby laying a foundation for subsequent studies applying ML approaches to TCM.

Flower Mapping in Grasslands With Drones and Deep Learning

Manual assessment of flower abundance of different flowering plant species in grasslands is a time-consuming process. We present an automated approach to determine the flower abundance in grasslands from drone-based aerial images by using deep learning (Faster R-CNN) object detection approach, which was trained and evaluated on data from five flights at two sites. Our deep learning network was able to identify and classify individual flowers. The novel method allowed generating spatially explicit maps of flower abundance that met or exceeded the accuracy of the manual-count-data extrapolation method while being less labor intensive. The results were very good for some types of flowers, with precision and recall being close to or higher than 90%. Other flowers were detected poorly due to reasons such as lack of enough training data, appearance changes due to phenology, or flowers being too small to be reliably distinguishable on the aerial images. The method was able to give precise estimates of the abundance of many flowering plant species. In the future, the collection of more training data will allow better predictions for the flowers that are not well predicted yet. The developed pipeline can be applied to any sort of aerial object detection problem.

Classifying Forest Structure of Red-Cockaded Woodpecker Habitat Using Structure from Motion Elevation Data De-Rived from sUAS Imagery

Small unmanned aerial systems (sUAS) and relatively new photogrammetry software solutions are creating opportunities for forest managers to perform spatial analysis more efficiently and cost-effectively. This study aims to identify a method for leveraging these technologies to analyze vertical forest structure of red-cockaded woodpecker habitat in Montgomery County, Texas. Traditional sampling methods would require numerous hours of ground surveying and data collection using various measuring techniques. Structure from Motion (SfM), a photogrammetric method for creating 3-D structure from 2-D images, provides an alternative to relatively expensive LIDAR sensing technologies and can accurately model the high level of complexity found within our study area’s vertical structure. DroneDeploy, a photogrammetry processing app service, was used to post-process and create a point cloud, which was later further processed into a Canopy Height Model (CHM). Using supervised, object-based classification and comparing multiple classifier algorithms, classifications maps were generated with a best overall accuracy of 84.8% using Support Vector Machine in ArcGIS Pro software. Appropriately sized training sample datasets, correctly processed elevation data, and proper image segmentation were among the major factors impacting classification accuracy during the numerous classification iterations performed.

Comparison of Classical Methods and Mask R-CNN for Automatic Tree Detection and Mapping Using UAV Imagery

Detecting and mapping individual trees accurately and automatically from remote sensing images is of great significance for precision forest management. Many algorithms, including classical methods and deep learning techniques, have been developed and applied for tree crown detection from remote sensing images. However, few studies have evaluated the accuracy of different individual tree detection (ITD) algorithms and their data and processing requirements. This study explored the accuracy of ITD using local maxima (LM) algorithm, marker-controlled watershed segmentation (MCWS), and Mask Region-based Convolutional Neural Networks (Mask R-CNN) in a young plantation forest with different test images. Manually delineated tree crowns from UAV imagery were used for accuracy assessment of the three methods, followed by an evaluation of the data processing and application requirements for three methods to detect individual trees. Overall, Mask R-CNN can best use the information in multi-band input images for detecting individual trees. The results showed that the Mask R-CNN model with the multi-band combination produced higher accuracy than the model with a single-band image, and the RGB band combination achieved the highest accuracy for ITD (F1 score = 94.68%). Moreover, the Mask R-CNN models with multi-band images are capable of providing higher accuracies for ITD than the LM and MCWS algorithms. The LM algorithm and MCWS algorithm also achieved promising accuracies for ITD when the canopy height model (CHM) was used as the test image (F1 score = 87.86% for LM algorithm, F1 score = 85.92% for MCWS algorithm). The LM and MCWS algorithms are easy to use and lower computer computational requirements, but they are unable to identify tree species and are limited by algorithm parameters, which need to be adjusted for each classification. It is highlighted that the application of deep learning with its end-to-end-learning approach is very efficient and capable of deriving the information from multi-layer images, but an additional training set is needed for model training, robust computer resources are required, and a large number of accurate training samples are necessary. This study provides valuable information for forestry practitioners to select an optimal approach for detecting individual trees.

Detection of Windthrown Tree Stems on UAV-Orthomosaics Using U-Net Convolutional Networks

The increasing number of severe storm events is threatening European forests. Besides the primary damages directly caused by storms, there are secondary damages such as bark beetle outbreaks and tertiary damages due to negative effects on the market. These subsequent damages can be minimized if a detailed overview of the affected area and the amount of damaged wood can be obtained quickly and included in the planning of clearance measures. The present work utilizes UAV-orthophotos and an adaptation of the U-Net architecture for the semantic segmentation and localization of windthrown stems. The network was pre-trained with generic datasets, randomly combining stems and background samples in a copy–paste augmentation, and afterwards trained with a specific dataset of a particular windthrow. The models pre-trained with generic datasets containing 10, 50 and 100 augmentations per annotated windthrown stems achieved F1-scores of 73.9% (S1Mod10), 74.3% (S1Mod50) and 75.6% (S1Mod100), outperforming the baseline model (F1-score 72.6%), which was not pre-trained. These results emphasize the applicability of the method to correctly identify windthrown trees and suggest the collection of training samples from other tree species and windthrow areas to improve the ability to generalize. Further enhancements of the network architecture are considered to improve the classification performance and to minimize the calculative costs.

Technical Workflow Development for Integrating Drone Surveys and Entomological Sampling to Characterise Aquatic Larval Habitats of Anopheles funestus in Agricultural Landscapes in Côte d'Ivoire

Land-use practices such as agriculture can impact mosquito vector breeding ecology, resulting in changes in disease transmission. The typical breeding habitats of Africa's second most important malaria vector Anopheles funestus are large, semipermanent water bodies, which make them potential candidates for targeted larval source management. This is a technical workflow for the integration of drone surveys and mosquito larval sampling, designed for a case study aiming to characterise An. funestus breeding sites near two villages in an agricultural setting in Côte d'Ivoire. Using satellite remote sensing data, we developed an environmentally and spatially representative sampling frame and conducted paired mosquito larvae and drone mapping surveys from June to August 2021. To categorise the drone imagery, we also developed a land cover classification scheme with classes relative to An. funestus breeding ecology. We sampled 189 potential breeding habitats, of which 119 (63%) were positive for the Anopheles genus and nine (4.8%) were positive for An. funestus. We mapped 30.42 km2 of the region of interest including all water bodies which were sampled for larvae. These data can be used to inform targeted vector control efforts, although its generalisability over a large region is limited by the fine-scale nature of this study area. This paper develops protocols for integrating drone surveys and statistically rigorous entomological sampling, which can be adjusted to collect data on vector breeding habitats in other ecological contexts. Further research using data collected in this study can enable the development of deep-learning algorithms for identifying An. funestus breeding habitats across rural agricultural landscapes in Côte d'Ivoire and the analysis of risk factors for these sites.

A nested parallel multiscale convolution for cerebrovascular segmentation

PURPOSE

Cerebrovascular segmentation in magnetic resonance imaging (MRI) plays an important role in the diagnosis and treatment of cerebrovascular diseases. Many segmentation frameworks based on convolutional neural networks (CNNs) or U-Net-like structures have been proposed for cerebrovascular segmentation. Unfortunately, the segmentation results are still unsatisfactory, particularly in the small/thin cerebrovascular due to the following reasons: (1) the lack of attention to multiscale features in encoder caused by the convolutions with single kernel size; (2) insufficient extraction of shallow and deep-seated features caused by the depth limitation of transmission path between encoder and decoder; (3) insufficient utilization of the extracted features in decoder caused by less attention to multiscale features.

METHODS

Inspired by U-Net++, we propose a novel 3D U-Net-like framework termed Usception for small cerebrovascular. It includes three blocks: Reduction block, Gap block, and Deep block, aiming to: (1) improve feature extraction ability by grouping different convolution sizes; (2) increase the number of multiscale features in different layers by grouping paths of different depths between encoder and decoder; (3) maximize the ability of decoder in recovering multiscale features from Reduction and Gap block by using convolutions with different kernel sizes.

RESULTS

The proposed framework is evaluated on three public and in-house clinical magnetic resonance angiography (MRA) data sets. The experimental results show that our framework reaches an average dice score of 69.29%, 87.40%, 77.77% on three data sets, which outperform existing state-of-the-art methods. We also validate the effectiveness of each block through ablation experiments.

CONCLUSIONS

By means of the combination of Inception-ResNet and dimension-expanded U-Net++, the proposed framework has demonstrated its capability to maximize multiscale feature extraction, thus achieving competitive segmentation results for small cerebrovascular.

Mauritia flexuosa palm trees airborne mapping with deep convolutional neural network

Accurately mapping individual tree species in densely forested environments is crucial to forest inventory. When considering only RGB images, this is a challenging task for many automatic photogrammetry processes. The main reason for that is the spectral similarity between species in RGB scenes, which can be a hindrance for most automatic methods. This paper presents a deep learning-based approach to detect an important multi-use species of palm trees (Mauritia flexuosa; i.e., Buriti) on aerial RGB imagery. In South-America, this palm tree is essential for many indigenous and local communities because of its characteristics. The species is also a valuable indicator of water resources, which comes as a benefit for mapping its location. The method is based on a Convolutional Neural Network (CNN) to identify and geolocate singular tree species in a high-complexity forest environment. The results returned a mean absolute error (MAE) of 0.75 trees and an F1-measure of 86.9%. These results are better than Faster R-CNN and RetinaNet methods considering equal experiment conditions. In conclusion, the method presented is efficient to deal with a high-density forest scenario and can accurately map the location of single species like the M. flexuosa palm tree and may be useful for future frameworks.

Deep neural networks based automated extraction of dugong feeding trails from UAV images in the intertidal seagrass beds

Dugongs (Dugong dugon) are seagrass specialists distributed in shallow coastal waters in tropical and subtropical seas. The area and distribution of the dugongs’ feeding trails, which are unvegetated winding tracks left after feeding, have been used as an indicator of their feeding ground utilization. However, current ground-based measurements of these trails require a large amount of time and effort. Here, we developed effective methods to observe the dugongs’ feeding trails using unmanned aerial vehicle (UAV) images (1) by extracting the dugong feeding trails using deep neural networks. Furthermore, we demonstrated two applications as follows; (2) extraction of the daily new feeding trails with deep neural networks and (3) estimation the direction of the feeding trails. We obtained aerial photographs from the intertidal seagrass bed at Talibong Island, Trang Province, Thailand. The F1 scores, which are a measure of binary classification model’s accuracy taking false positives and false negatives into account, for the method (1) were 89.5% and 87.7% for the images with ground sampling resolutions of 1 cm/pixel and 0.5 cm/pixel, respectively, while the F1 score for the method (2) was 61.9%. The F1 score for the method (1) was high enough to perform scientific studies on the dugong. However, the method (2) should be improved, and there remains a need for manual correction. The mean area of the extracted daily new feeding trails from September 12–27, 2019, was 187.8 m² per day (n = 9). Total 63.9% of the feeding trails was estimated to have direction within a range of 112.5° and 157.5°. These proposed new methods will reduce the time and efforts required for future feeding trail observations and contribute to future assessments of the dugongs’ seagrass habitat use.

Deep learning and citizen science enable automated plant trait predictions from photographs

Plant functional traits ('traits') are essential for assessing biodiversity and ecosystem processes, but cumbersome to measure. To facilitate trait measurements, we test if traits can be predicted through visible morphological features by coupling heterogeneous photographs from citizen science (iNaturalist) with trait observations (TRY database) through Convolutional Neural Networks (CNN). Our results show that image features suffice to predict several traits representing the main axes of plant functioning. The accuracy is enhanced when using CNN ensembles and incorporating prior knowledge on trait plasticity and climate. Our results suggest that these models generalise across growth forms, taxa and biomes around the globe. We highlight the applicability of this approach by producing global trait maps that reflect known macroecological patterns. These findings demonstrate the potential of Big Data derived from professional and citizen science in concert with CNN as powerful tools for an efficient and automated assessment of Earth's plant functional diversity.

Soil Salinity Inversion in Coastal Corn Planting Areas by the Satellite-UAV-Ground Integration Approach

Soil salinization is a significant factor affecting corn growth in coastal areas. How to use multi-source remote sensing data to achieve the target of rapid, efficient and accurate soil salinity monitoring in a large area is worth further study. In this research, using Kenli District of the Yellow River Delta as study area, the inversion of soil salinity in a corn planting area was carried out based on the integration of ground imaging hyperspectral, unmanned aerial vehicles (UAV) multispectral and Sentinel-2A satellite multispectral images. The UAV and ground images were fused, and the partial least squares inversion model was constructed by the fused UAV image. Then, inversion model was scaled up to the satellite by the TsHARP method, and finally, the accuracy of the satellite-UAV-ground inversion model and results was verified. The results show that the band fusion of UAV and ground images effectively enrich the spectral information of the UAV image. The accuracy of the inversion model constructed based on the fused UAV images was improved. The inversion results of soil salinity based on the integration of satellite-UAV-ground were highly consistent with the measured soil salinity (R2 = 0.716 and RMSE = 0.727), and the inversion model had excellent universal applicability. This research integrated the advantages of multi-source data to establish a unified satellite-UAV-ground model, which improved the ability of large-scale remote sensing data to finely indicate soil salinity.

Style transfer-based domain adaptation for vegetation segmentation with optical imagery

Style transfer methods are an important task for domain adaptation of optical imagery to improve the performance of deep learning models when using different sensor systems. For the transformation between datasets, cycle-consistent adversarial networks achieve good results. However, during the style transfer process, characteristic spectral information that is essential for the analysis of vegetation could get lost. This issue is especially important since optical airborne- and spaceborne-based sensors are frequently used to investigate vegetation ground coverage and its condition. In this paper, we present a cycle-consistent adversarial domain adaptation method with four input channels for the segmentation of vegetation areas using index-based metrics. We show that our method preserves the specific ratio between the near-IR and RGB bands and improves the segmentation network performance for the target domain.

Deep Learning in Forestry Using UAV-Acquired RGB Data: A Practical Review

Forests are the planet’s main CO2 filtering agent as well as important economical, environmental and social assets. Climate change is exerting an increased stress, resulting in a need for improved research methodologies to study their health, composition or evolution. Traditionally, information about forests has been collected using expensive and work-intensive field inventories, but in recent years unoccupied autonomous vehicles (UAVs) have become very popular as they represent a simple and inexpensive way to gather high resolution data of large forested areas. In addition to this trend, deep learning (DL) has also been gaining much attention in the field of forestry as a way to include the knowledge of forestry experts into automatic software pipelines tackling problems such as tree detection or tree health/species classification. Among the many sensors that UAVs can carry, RGB cameras are fast, cost-effective and allow for straightforward data interpretation. This has resulted in a large increase in the amount of UAV-acquired RGB data available for forest studies. In this review, we focus on studies that use DL and RGB images gathered by UAVs to solve practical forestry research problems. We summarize the existing studies, provide a detailed analysis of their strengths paired with a critical assessment on common methodological problems and include other information, such as available public data and code resources that we believe can be useful for researchers that want to start working in this area. We structure our discussion using three main families of forestry problems: (1) individual Tree Detection, (2) tree Species Classification, and (3) forest Anomaly Detection (forest fires and insect Infestation).

Deep Convolutional Neural Network for Large-Scale Date Palm Tree Mapping from UAV-Based Images

Large-scale mapping of date palm trees is vital for their consistent monitoring and sustainable management, considering their substantial commercial, environmental, and cultural value. This study presents an automatic approach for the large-scale mapping of date palm trees from very-high-spatial-resolution (VHSR) unmanned aerial vehicle (UAV) datasets, based on a deep learning approach. A U-Shape convolutional neural network (U-Net), based on a deep residual learning framework, was developed for the semantic segmentation of date palm trees. A comprehensive set of labeled data was established to enable the training and evaluation of the proposed segmentation model and increase its generalization capability. The performance of the proposed approach was compared with those of various state-of-the-art fully convolutional networks (FCNs) with different encoder architectures, including U-Net (based on VGG-16 backbone), pyramid scene parsing network, and two variants of DeepLab V3+. Experimental results showed that the proposed model outperformed other FCNs in the validation and testing datasets. The generalizability evaluation of the proposed approach on a comprehensive and complex testing dataset exhibited higher classification accuracy and showed that date palm trees could be automatically mapped from VHSR UAV images with an F-score, mean intersection over union, precision, and recall of 91%, 85%, 0.91, and 0.92, respectively. The proposed approach provides an efficient deep learning architecture for the automatic mapping of date palm trees from VHSR UAV-based images.

Equivalent-input-disturbance estimator-based event-triggered control design for master-slave neural networks

This paper investigates the robust synchronization problem for a class of master-slave neural networks (MSNNs) subject to network-induced delays, unknown time-varying uncertainty, and exogenous disturbances. An equivalent-input-disturbance (EID) estimation technique is applied to compensate for the effects of unknown uncertainty and disturbances in the system output. In addition, to reduce the burden of the communication channel in the addressed MSNNs and improve the utilization of bandwidth an event-triggered control protocol is developed to obtain the synchronization of MSNNs. In particular, event-triggering conditions are verified periodically at every sampling instant in both sensors and actuators to avoid the Zeno behavior in the networks. By designing an appropriate low-pass filter in the EID estimator block, the accuracy of disturbance estimation performance is improved. Moreover, by concatenating the synchronization error, observer, and filter states as a single state vector, an augmented system is formulated. Then the tangible delay-dependent stability condition for that augmented system is established by employing the Lyapunov stability theory and reciprocally convex approach. Based on the feasible solutions of the derived stability conditions, the event-triggering parameters, controller, and observer gains are co-designed. Finally, two toy examples are given to illustrate the established theoretical findings.

A Farm Management Information System for Semi-Supervised Path Planning and Autonomous Vehicle Control

This paper presents a farm management information system targeting improvements in the ease of use and sustainability of robot farming systems. The system integrates the functionalities of field survey, path planning, monitoring, and controlling agricultural vehicles in real time. Firstly, a Grabcut-based semi-supervised field registration method is proposed for arable field detection from the orthoimage taken by the drone with an RGB camera. It partitions a complex field into simple geometric entities with simple user interaction. The average Mean Intersection over Union is about 0.95 when the field size ranges from 2.74 ha to 5.06 ha. In addition, a desktop software and a web application are developed as the entity of an FMIS. Compared to existing FMISs, this system provides more advanced features in robot farming, while providing simpler user interaction and better results. It allows clients to invoke web services and receive responses independent of programming language and platforms. Moreover, the system is compatible with other services, users, and devices following the open-source access protocol. We have evaluated the system by controlling 5 robot tractors with a 2 Hz communication frequency. The communication protocols will be publicly available to protentional users.

Early Detection of Encroaching Woody Juniperus virginiana and Its Classification in Multi-Species Forest Using UAS Imagery and Semantic Segmentation Algorithms

Woody plant encroachment into grasslands ecosystems causes significantly ecological destruction and economic losses. Effective and efficient management largely benefits from accurate and timely detection of encroaching species at an early development stage. Recent advances in unmanned aircraft systems (UAS) enabled easier access to ultra-high spatial resolution images at a centimeter level, together with the latest machine learning based image segmentation algorithms, making it possible to detect small-sized individuals of target species at early development stage and identify them when mixed with other species. However, few studies have investigated the optimal practical spatial resolution of early encroaching species detection. Hence, we investigated the performance of four popular semantic segmentation algorithms (decision tree, DT; random forest, RF; AlexNet; and ResNet) on a multi-species forest classification case with UAS-collected RGB images in original and down-sampled coarser spatial resolutions. The objective of this study was to explore the optimal segmentation algorithm and spatial resolution for eastern redcedar (Juniperus virginiana, ERC) early detection and its classification within a multi-species forest context. To be specific, firstly, we implemented and compared the performance of the four semantic segmentation algorithms with images in the original spatial resolution (0.694 cm). The highest overall accuracy was 0.918 achieved by ResNet with a mean interaction over union at 85.0%. Secondly, we evaluated the performance of ResNet algorithm with images in down-sampled spatial resolutions (1 cm to 5 cm with 0.5 cm interval). When applied on the down-sampled images, ERC segmentation performance decreased with decreasing spatial resolution, especially for those images coarser than 3 cm spatial resolution. The UAS together with the state-of-the-art semantic segmentation algorithms provides a promising tool for early-stage detection and localization of ERC and the development of effective management strategies for mixed-species forest management.

Method for accurate multi-growth-stage estimation of fractional vegetation cover using unmanned aerial vehicle remote sensing

BACKGROUND

Fractional vegetation cover (FVC) is an important parameter for evaluating crop-growth status. Optical remote-sensing techniques combined with the pixel dichotomy model (PDM) are widely used to estimate cropland FVC with medium to high spatial resolution on the ground. However, PDM-based FVC estimation is limited by effects stemming from the variation of crop canopy chlorophyll content (CCC). To overcome this difficulty, we propose herein a "fan-shaped method" (FSM) that uses a CCC spectral index (SI) and a vegetation SI to create a two-dimensional scatter map in which the three vertices represent high-CCC vegetation, low-CCC vegetation, and bare soil. The FVC at each pixel is determined based on the spatial location of the pixel in the two-dimensional scatter map, which mitigates the effects of CCC on the PDM. To evaluate the accuracy of FSM estimates of the FVC, we analyze the spectra obtained from (a) the PROSAIL model and (b) a spectrometer mounted on an unmanned aerial vehicle platform. Specifically, we use both the proposed FSM and traditional remote-sensing FVC-estimation methods (both linear and nonlinear regression and PDM) to estimate soybean FVC.

RESULTS

Field soybean CCC measurements indicate that (a) the soybean CCC increases continuously from the flowering growth stage to the later-podding growth stage, and then decreases with increasing crop growth stages, (b) the coefficient of variation of soybean CCC is very large in later growth stages (31.58-35.77%) and over all growth stages (26.14%). FVC samples with low CCC are underestimated by the PDM. Linear and nonlinear regression underestimates (overestimates) FVC samples with low (high) CCC. The proposed FSM depends less on CCC and is thus a robust method that can be used for multi-stage FVC estimation of crops with strongly varying CCC.

CONCLUSIONS

Estimates and maps of FVC based on the later growth stages and on multiple growth stages should consider the variation of crop CCC. FSM can mitigates the effect of CCC by conducting a PDM at each CCC level. The FSM is a robust method that can be used to estimate FVC based on multiple growth stages where crop CCC varies greatly.

Deep Learning and Phenology Enhance Large-Scale Tree Species Classification in Aerial Imagery during a Biosecurity Response

The ability of deep convolutional neural networks (deep learning) to learn complex visual characteristics offers a new method to classify tree species using lower-cost data such as regional aerial RGB imagery. In this study, we use 10 cm resolution imagery and 4600 trees to develop a deep learning model to identify Metrosideros excelsa (pōhutukawa)—a culturally important New Zealand tree that displays distinctive red flowers during summer and is under threat from the invasive pathogen Austropuccinia psidii (myrtle rust). Our objectives were to compare the accuracy of deep learning models that could learn the distinctive visual characteristics of the canopies with tree-based models (XGBoost) that used spectral and textural metrics. We tested whether the phenology of pōhutukawa could be used to enhance classification by using multitemporal aerial imagery that showed the same trees with and without widespread flowering. The XGBoost model achieved an accuracy of 86.7% on the dataset with strong phenology (flowering). Without phenology, the accuracy fell to 79.4% and the model relied on the blueish hue and texture of the canopies. The deep learning model achieved 97.4% accuracy with 96.5% sensitivity and 98.3% specificity when leveraging phenology—even though the intensity of flowering varied substantially. Without strong phenology, the accuracy of the deep learning model remained high at 92.7% with sensitivity of 91.2% and specificity of 94.3% despite significant variation in the appearance of non-flowering pōhutukawa. Pooling time-series imagery did not enhance either approach. The accuracy of XGBoost and deep learning models were, respectively, 83.2% and 95.2%, which were of intermediate precision between the separate models.

Learning-based defect recognition for quasi-periodic HRSTEM images

Controlling crystalline material defects is crucial, as they affect properties of the material that may be detrimental or beneficial for the final performance of a device. Defect analysis on the sub-nanometer scale is enabled by high-resolution scanning transmission electron microscopy (HRSTEM), where the identification of defects is currently carried out based on human expertise. However, the process is tedious, highly time consuming and, in some cases, yields ambiguous results. Here we propose a semi-supervised machine learning method that assists in the detection of lattice defects from atomic resolution HRSTEM images. It involves a convolutional neural network that classifies image patches as defective or non-defective, a graph-based heuristic that chooses one non-defective patch as a model, and finally an automatically generated convolutional filter bank, which highlights symmetry breaking such as stacking faults, twin defects and grain boundaries. Additionally, we suggest a variance filter to segment amorphous regions and beam defects. The algorithm is tested on III-V/Si crystalline materials and successfully evaluated against different metrics and a baseline approach, showing promising results even for extremely small training data sets and for noise compromised images. By combining the data-driven classification generality, robustness and speed of deep learning with the effectiveness of image filters in segmenting faulty symmetry arrangements, we provide a valuable open-source tool to the microscopist community that can streamline future HRSTEM analyses of crystalline materials.

Repeatable Semantic Reef-Mapping through Photogrammetry and Label-Augmentation

In an endeavor to study natural systems at multiple spatial and taxonomic resolutions, there is an urgent need for automated, high-throughput frameworks that can handle plethora of information. The coalescence of remote-sensing, computer-vision, and deep-learning elicits a new era in ecological research. However, in complex systems, such as marine-benthic habitats, key ecological processes still remain enigmatic due to the lack of cross-scale automated approaches (mms to kms) for community structure analysis. We address this gap by working towards scalable and comprehensive photogrammetric surveys, tackling the profound challenges of full semantic segmentation and 3D grid definition. Full semantic segmentation (where every pixel is classified) is extremely labour-intensive and difficult to achieve using manual labeling. We propose using label-augmentation, i.e., propagation of sparse manual labels, to accelerate the task of full segmentation of photomosaics. Photomosaics are synthetic images generated from a projected point-of-view of a 3D model. In the lack of navigation sensors (e.g., a diver-held camera), it is difficult to repeatably determine the slope-angle of a 3D map. We show this is especially important in complex topographical settings, prevalent in coral-reefs. Specifically, we evaluate our approach on benthic habitats, in three different environments in the challenging underwater domain. Our approach for label-augmentation shows human-level accuracy in full segmentation of photomosaics using labeling as sparse as 0.1%, evaluated on several ecological measures. Moreover, we found that grid definition using a leveler improves the consistency in community-metrics obtained due to occlusions and topology (angle and distance between objects), and that we were able to standardise the 3D transformation with two percent error in size measurements. By significantly easing the annotation process for full segmentation and standardizing the 3D grid definition we present a semantic mapping methodology enabling change-detection, which is practical, swift, and cost-effective. Our workflow enables repeatable surveys without permanent markers and specialized mapping gear, useful for research and monitoring, and our code is available online. Additionally, we release the Benthos data-set, fully manually labeled photomosaics from three oceanic environments with over 4500 segmented objects useful for research in computer-vision and marine ecology.

Study of Quantized Hardware Deep Neural Networks Based on Resistive Switching Devices, Conventional versus Convolutional Approaches

A comprehensive analysis of two types of artificial neural networks (ANN) is performed to assess the influence of quantization on the synaptic weights. Conventional multilayer-perceptron (MLP) and convolutional neural networks (CNN) have been considered by changing their features in the training and inference contexts, such as number of levels in the quantization process, the number of hidden layers on the network topology, the number of neurons per hidden layer, the image databases, the number of convolutional layers, etc. A reference technology based on 1T1R structures with bipolar memristors including HfO2 dielectrics was employed, accounting for different multilevel schemes and the corresponding conductance quantization algorithms. The accuracy of the image recognition processes was studied in depth. This type of studies are essential prior to hardware implementation of neural networks. The obtained results support the use of CNNs for image domains. This is linked to the role played by convolutional layers at extracting image features and reducing the data complexity. In this case, the number of synaptic weights can be reduced in comparison to MLPs.

New tools for old problems - comparing drone- and field-based assessments of a problematic plant species

Plant species that negatively affect their environment by encroachment require constant management and monitoring through field surveys. Drones have been suggested to support field surveyors allowing more accurate mapping with just-in-time aerial imagery. Furthermore, object-based image analysis tools could increase the accuracy of species maps. However, only few studies compare species distribution maps resulting from traditional field surveys and object-based image analysis using drone imagery. We acquired drone imagery for a saltmarsh area (18 ha) on the Hallig Nordstrandischmoor (Germany) with patches of Elymus athericus, a tall grass which encroaches higher parts of saltmarshes. A field survey was conducted afterwards using the drone orthoimagery as a baseline. We used object-based image analysis (OBIA) to segment CIR imagery into polygons which were classified into eight land cover classes. Finally, we compared polygons of the field-based and OBIA-based maps visually and for location, area, and overlap before and after post-processing. OBIA-based classification yielded good results (kappa = 0.937) and agreed in general with the field-based maps (field = 6.29 ha, drone = 6.22 ha with E. athericus dominance). Post-processing revealed 0.31 ha of misclassified polygons, which were often related to water runnels or shadows, leaving 5.91 ha of E. athericus cover. Overlap of both polygon maps was only 70% resulting from many small patches identified where E. athericus was absent. In sum, drones can greatly support field surveys in monitoring of plant species by allowing for accurate species maps and just-in-time captured very-high-resolution imagery.

Using U-Net-Like Deep Convolutional Neural Networks for Precise Tree Recognition in Very High Resolution RGB (Red, Green, Blue) Satellite Images

Very high resolution satellite imageries provide an excellent foundation for precise mapping of plant communities and even single plants. We aim to perform individual tree recognition on the basis of very high resolution RGB (red, green, blue) satellite images using deep learning approaches for northern temperate mixed forests in the Primorsky Region of the Russian Far East. We used a pansharpened satellite RGB image by GeoEye-1 with a spatial resolution of 0.46 m/pixel, obtained in late April 2019. We parametrized the standard U-Net convolutional neural network (CNN) and trained it in manually delineated satellite images to solve the satellite image segmentation problem. For comparison purposes, we also applied standard pixel-based classification algorithms, such as random forest, k-nearest neighbor classifier, naive Bayes classifier, and quadratic discrimination. Pattern-specific features based on grey level co-occurrence matrices (GLCM) were computed to improve the recognition ability of standard machine learning methods. The U-Net-like CNN allowed us to obtain precise recognition of Mongolian poplar (Populus suaveolens Fisch. ex Loudon s.l.) and evergreen coniferous trees (Abies holophylla Maxim., Pinus koraiensis Siebold & Zucc.). We were able to distinguish species belonging to either poplar or coniferous groups but were unable to separate species within the same group (i.e. A. holophylla and P. koraiensis were not distinguishable). The accuracy of recognition was estimated by several metrics and exceeded values obtained for standard machine learning approaches. In contrast to pixel-based recognition algorithms, the U-Net-like CNN does not lead to an increase in false-positive decisions when facing green-colored objects that are similar to trees. By means of U-Net-like CNN, we obtained a mean accuracy score of up to 0.96 in our computational experiments. The U-Net-like CNN recognizes tree crowns not as a set of pixels with known RGB intensities but as spatial objects with a specific geometry and pattern. This CNN’s specific feature excludes misclassifications related to objects of similar colors as objects of interest. We highlight that utilization of satellite images obtained within the suitable phenological season is of high importance for successful tree recognition. The suitability of the phenological season is conceptualized as a group of conditions providing highlighting objects of interest over other components of vegetation cover. In our case, the use of satellite images captured in mid-spring allowed us to recognize evergreen fir and pine trees as the first class of objects (“conifers”) and poplars as the second class, which were in a leafless state among other deciduous tree species.

Identifying Soil Erosion Processes in Alpine Grasslands on Aerial Imagery with a U-Net Convolutional Neural Network

Erosion in alpine grasslands is a major threat to ecosystem services of alpine soils. Natural causes for the occurrence of soil erosion are steep topography and prevailing climate conditions in combination with soil fragility. To increase our understanding of ongoing erosion processes and support sustainable land-use management, there is a need to acquire detailed information on spatial occurrence and temporal trends. Existing approaches to identify these trends are typically laborious, have lack of transferability to other regions, and are consequently only applicable to smaller regions. In order to overcome these limitations and create a sophisticated erosion monitoring tool capable of large-scale analysis, we developed a model based on U-Net, a fully convolutional neural network, to map different erosion processes on high-resolution aerial images (RGB, 0.25–0.5 m). U-Net was trained on a high-quality data set consisting of labeled erosion sites mapped with object-based image analysis (OBIA) for the Urseren Valley (Central Swiss Alps) for five aerial images (16 year period). We used the U-Net model to map the same study area and conduct quality assessments based on a held-out test region and a temporal transferability test on new images. Erosion classes are assigned according to their type (shallow landslide and sites with reduced vegetation affected by sheet erosion) or land-use impacts (livestock trails and larger management affected areas). We show that results obtained by OBIA and U-Net follow similar linear trends for the 16 year study period, exhibiting increases in total degraded area of 167% and 201%, respectively. Segmentations of eroded sites are generally in good agreement, but also display method-specific differences, which lead to an overall precision of 73%, a recall of 84%, and a F1-score of 78%. Our results show that U-Net is transferable to spatially (within our study area) and temporally unseen data (data from new years) and is therefore a method suitable to efficiently and successfully capture the temporal trends and spatial heterogeneity of degradation in alpine grasslands. Additionally, U-Net is a powerful and robust tool to map erosion sites in a predictive manner utilising large amounts of new aerial imagery.

Machine learning for landslides prevention: a survey

Landslides are one of the most critical categories of natural disasters worldwide and induce severely destructive outcomes to human life and the overall economic system. To reduce its negative effects, landslides prevention has become an urgent task, which includes investigating landslide-related information and predicting potential landslides. Machine learning is a state-of-the-art analytics tool that has been widely used in landslides prevention. This paper presents a comprehensive survey of relevant research on machine learning applied in landslides prevention, mainly focusing on (1) landslides detection based on images, (2) landslides susceptibility assessment, and (3) the development of landslide warning systems. Moreover, this paper discusses the current challenges and potential opportunities in the application of machine learning algorithms for landslides prevention.

Semantic Segmentation of Airborne LiDAR Data in Maya Archaeology

Airborne LiDAR produced large amounts of data for archaeological research over the past decade. Labeling this type of archaeological data is a tedious process. We used a data set from Pacunam LiDAR Initiative survey of lowland Maya region in Guatemala. The data set contains ancient Maya structures that were manually labeled, and ground verified to a large extent. We have built and compared two deep learning-based models, U-Net and Mask R-CNN, for semantic segmentation. The segmentation models were used in two tasks: identification of areas of ancient construction activity, and identification of the remnants of ancient Maya buildings. The U-Net based model performed better in both tasks and was capable of correctly identifying 60–66% of all objects, and 74–81% of medium sized objects. The quality of the resulting prediction was evaluated using a variety of quantifiers. Furthermore, we discuss the problems of re-purposing the archaeological style labeling for production of valid machine learning training sets. Ultimately, we outline the value of these models for archaeological research and present the road map to produce a useful decision support system for recognition of ancient objects in LiDAR data.

An Agave Counting Methodology Based on Mathematical Morphology and Images Acquired through Unmanned Aerial Vehicles

Blue agave is an important commercial crop in Mexico, and it is the main source of the traditional mexican beverage known as tequila. The variety of blue agave crop known as Tequilana Weber is a crucial element for tequila agribusiness and the agricultural economy in Mexico. The number of agave plants in the field is one of the main parameters for estimating production of tequila. In this manuscript, we describe a mathematical morphology-based algorithm that addresses the agave automatic counting task. The proposed methodology was applied to a set of real images collected using an Unmanned Aerial Vehicle equipped with a digital Red-Green-Blue (RGB) camera. The number of plants automatically identified in the collected images was compared to the number of plants counted by hand. Accuracy of the proposed algorithm depended on the size heterogeneity of plants in the field and illumination. Accuracy ranged from 0.8309 to 0.9806, and performance of the proposed algorithm was satisfactory.

Object Detection and Image Segmentation with Deep Learning on Earth Observation Data: A Review—Part II: Applications

In Earth observation (EO), large-scale land-surface dynamics are traditionally analyzed by investigating aggregated classes. The increase in data with a very high spatial resolution enables investigations on a fine-grained feature level which can help us to better understand the dynamics of land surfaces by taking object dynamics into account. To extract fine-grained features and objects, the most popular deep-learning model for image analysis is commonly used: the convolutional neural network (CNN). In this review, we provide a comprehensive overview of the impact of deep learning on EO applications by reviewing 429 studies on image segmentation and object detection with CNNs. We extensively examine the spatial distribution of study sites, employed sensors, used datasets and CNN architectures, and give a thorough overview of applications in EO which used CNNs. Our main finding is that CNNs are in an advanced transition phase from computer vision to EO. Upon this, we argue that in the near future, investigations which analyze object dynamics with CNNs will have a significant impact on EO research. With a focus on EO applications in this Part II, we complete the methodological review provided in Part I.

Tree Crown Delineation Algorithm Based on a Convolutional Neural Network

Tropical forests concentrate the largest diversity of species on the planet and play a key role in maintaining environmental processes. Due to the importance of those forests, there is growing interest in mapping their components and getting information at an individual tree level to conduct reliable satellite-based forest inventory for biomass and species distribution qualification. Individual tree crown information could be manually gathered from high resolution satellite images; however, to achieve this task at large-scale, an algorithm to identify and delineate each tree crown individually, with high accuracy, is a prerequisite. In this study, we propose the application of a convolutional neural network—Mask R-CNN algorithm—to perform the tree crown detection and delineation. The algorithm uses very high-resolution satellite images from tropical forests. The results obtained are promising—the R e c a l l , P r e c i s i o n , and F 1 score values obtained were were 0.81 , 0.91 , and 0.86 , respectively. In the study site, the total of tree crowns delineated was 59,062 . These results suggest that this algorithm can be used to assist the planning and conduction of forest inventories. As the algorithm is based on a Deep Learning approach, it can be systematically trained and used for other regions.

Identification of different species of Zanthoxyli Pericarpium based on convolution neural network

Zanthoxyli Pericarpium (ZP) are the dried ripe peel of Zanthoxylum schinifolium Sieb. et Zucc (ZC) or Zanthoxylum bungeanum Maxim (ZB). It has wide range of uses both medicine and food, and favorable market value. The diverse specifications of components of ZP is exceptional, and the common aims of adulteration for economic profit is conducted. In this work, a novel method for the identification different species of ZP is proposed using convolutional neural networks (CNNs). The data used for the experiment is 5 classes obtained from camera and mobile phones. Firstly, the data considering 2 categories are trained to detect the labels by YOLO. Then, the multiple deep learning including VGG, ResNet, Inception v4, and DenseNet are introduced to identify the different species of ZP (HZB, DZB, OZB, ZA and JZC). In order to assess the performance of CNNs, compared with two traditional identification models including Support Vector Machines (SVM) and Back Propagation (BP). The experimental results demonstrate that the CNN model have a better performance to identify different species of ZP and the highest identification accuracy is 99.35%. The present study is proved to be a useful strategy for the discrimination of different traditional Chinese medicines (TCMs).

Automatic Windthrow Detection Using Very-High-Resolution Satellite Imagery and Deep Learning

Wind disturbances are significant phenomena in forest spatial structure and succession dynamics. They cause changes in biodiversity, impact on forest ecosystems at different spatial scales, and have a strong influence on economics and human beings. The reliable recognition and mapping of windthrow areas are of high importance from the perspective of forest management and nature conservation. Recent research in artificial intelligence and computer vision has demonstrated the incredible potential of neural networks in addressing image classification problems. The most efficient algorithms are based on artificial neural networks of nested and complex architecture (e.g., convolutional neural networks (CNNs)), which are usually referred to by a common term—deep learning. Deep learning provides powerful algorithms for the precise segmentation of remote sensing data. We developed an algorithm based on a U-Net-like CNN, which was trained to recognize windthrow areas in Kunashir Island, Russia. We used satellite imagery of very-high spatial resolution (0.5 m/pixel) as source data. We performed a grid search among 216 parameter combinations defining different U-Net-like architectures. The best parameter combination allowed us to achieve an overall accuracy for recognition of windthrow sites of up to 94% for forested landscapes by coniferous and mixed coniferous forests. We found that the false-positive decisions of our algorithm correspond to either seashore logs, which may look similar to fallen tree trunks, or leafless forest stands. While the former can be rectified by applying a forest mask, the latter requires the usage of additional information, which is not always provided by satellite imagery.