An End to End Process Development for UAV-SfM Based Forest Monitoring: Individual Tree Detection, Species Classification and Carbon Dynamics Simulation

UAV-Borne Imagery Can Supplement Airborne Lidar in the Precise Description of Dynamically Changing Shrubland Woody Vegetation

Airborne laser scanning (ALS) is increasingly used for detailed vegetation structure mapping; however, there are many local-scale applications where it is economically ineffective or unfeasible from the temporal perspective. Unmanned aerial vehicles (UAVs) or airborne imagery (AImg) appear to be promising alternatives, but only a few studies have examined this assumption outside economically exploited areas (forests, orchards, etc.). The main aim of this study was to compare the usability of normalized digital surface models (nDSMs) photogrammetrically derived from UAV-borne and airborne imagery to those derived from low- (1–2 pts/m2) and high-density (ca. 20 pts/m2) ALS-scanning for the precise local-scale modelling of woody vegetation structures (the number and height of trees/shrubs) across six dynamically changing shrubland sites. The success of the detection of woody plant tops was initially almost 100% for UAV-based models; however, deeper analysis revealed that this was due to the fact that omission and commission errors were approximately equal and the real accuracy was approx. 70% for UAV-based models compared to 95.8% for the high-density ALS model. The percentage mean absolute errors (%MAE) of shrub/tree heights derived from UAV data ranged between 12.2 and 23.7%, and AImg height accuracy was relatively lower (%MAE: 21.4–47.4). Combining UAV-borne or AImg-based digital surface models (DSM) with ALS-based digital terrain models (DTMs) significantly improved the nDSM height accuracy (%MAE: 9.4–13.5 and 12.2–25.0, respectively) but failed to significantly improve the detection of the number of individual shrubs/trees. The height accuracy and detection success using low- or high-density ALS did not differ. Therefore, we conclude that UAV-borne imagery has the potential to replace custom ALS in specific local-scale applications, especially at dynamically changing sites where repeated ALS is costly, and the combination of such data with (albeit outdated and sparse) ALS-based digital terrain models can further improve the success of the use of such data.

Multi-Species Individual Tree Segmentation and Identification Based on Improved Mask R-CNN and UAV Imagery in Mixed Forests

High-resolution UAV imagery paired with a convolutional neural network approach offers significant advantages in accurately measuring forestry ecosystems. Despite numerous studies existing for individual tree crown delineation, species classification, and quantity detection, the comprehensive situation in performing the above tasks simultaneously has rarely been explored, especially in mixed forests. In this study, we propose a new method for individual tree segmentation and identification based on the improved Mask R-CNN. For the optimized network, the fusion type in the feature pyramid network is modified from down-top to top-down to shorten the feature acquisition path among the different levels. Meanwhile, a boundary-weighted loss module is introduced to the cross-entropy loss function Lmask to refine the target loss. All geometric parameters (contour, the center of gravity and area) associated with canopies ultimately are extracted from the mask by a boundary segmentation algorithm. The results showed that F1-score and mAP for coniferous species were higher than 90%, and that of broadleaf species were located between 75%–85.44%. The producer’s accuracy of coniferous forests was distributed between 0.8–0.95 and that of broadleaf ranged in 0.87–0.93; user’s accuracy of coniferous was distributed between 0.81–0.84 and that of broadleaf ranged in 0.71–0.76. The total number of trees predicted was 50,041 for the entire study area, with an overall error of 5.11%. The method under study is compared with other networks including U-net and YOLOv3. Results in this study show that the improved Mask R-CNN has more advantages in broadleaf canopy segmentation and number detection.

Logging Trail Segmentation via a Novel U-Net Convolutional Neural Network and High-Density Laser Scanning Data

Logging trails are one of the main components of modern forestry. However, spotting the accurate locations of old logging trails through common approaches is challenging and time consuming. This study was established to develop an approach, using cutting-edge deep-learning convolutional neural networks and high-density laser scanning data, to detect logging trails in different stages of commercial thinning, in Southern Finland. We constructed a U-Net architecture, consisting of encoder and decoder paths with several convolutional layers, pooling and non-linear operations. The canopy height model (CHM), digital surface model (DSM), and digital elevation models (DEMs) were derived from the laser scanning data and were used as image datasets for training the model. The labeled dataset for the logging trails was generated from different references as well. Three forest areas were selected to test the efficiency of the algorithm that was developed for detecting logging trails. We designed 21 routes, including 390 samples of the logging trails and non-logging trails, covering all logging trails inside the stands. The results indicated that the trained U-Net using DSM (k = 0.846 and IoU = 0.867) shows superior performance over the trained model using CHM (k = 0.734 and IoU = 0.782), DEMavg (k = 0.542 and IoU = 0.667), and DEMmin (k = 0.136 and IoU = 0.155) in distinguishing logging trails from non-logging trails. Although the efficiency of the developed approach in young and mature stands that had undergone the commercial thinning is approximately perfect, it needs to be improved in old stands that have not received the second or third commercial thinning.

The current and future uses of machine learning in ecosystem service research

Machine learning (ML) expands traditional data analysis and presents a range of opportunities in ecosystem service (ES) research, offering rapid processing of 'big data' and enabling significant advances in data description and predictive modelling. Descriptive ML techniques group data with little or no prior domain specific assumptions; they can generate hypotheses and automatically sort data prior to other analyses. Predictive ML techniques allow for the predictive modelling of highly non-linear systems where casual mechanisms are poorly understood, as is often the case for ES. We conducted a review to explore how ML is used in ES research and to identify and quantify trends in the different ML approaches that are used. We reviewed 308 peer-reviewed publications and identified that ES studies implemented machine learning techniques in data description (64%; n = 308) and predictive modelling (44%), with some papers containing both categories. Classification and Regression Trees were the most popular techniques (60%), but unsupervised learning techniques were also used for descriptive tasks such as clustering to group or split data without prior assumptions (19%). Whilst there are examples of ES publications that apply ML with rigour, many studies do not have robust or repeatable methods. Some studies fail to report model settings (43%) or software used (28%), and many studies do not report carrying out any form of model hyperparameter tuning (67%) or test model generalisability (59%). Whilst studies use ML to analyse very large and complex datasets, ES research is generally not taking full advantage of the capacity of ML to model big data (1138 medium number of data points; 13 median quantity of variables). There is great further opportunity to utilise ML in ES research, to make better use of big data and to develop detailed modelling of spatial-temporal dynamics that meet stakeholder demands.

Unimodal and Multimodal Perception for Forest Management: Review and Dataset

Robotics navigation and perception for forest management are challenging due to the existence of many obstacles to detect and avoid and the sharp illumination changes. Advanced perception systems are needed because they can enable the development of robotic and machinery solutions to accomplish a smarter, more precise, and sustainable forestry. This article presents a state-of-the-art review about unimodal and multimodal perception in forests, detailing the current developed work about perception using a single type of sensors (unimodal) and by combining data from different kinds of sensors (multimodal). This work also makes a comparison between existing perception datasets in the literature and presents a new multimodal dataset, composed by images and laser scanning data, as a contribution for this research field. Lastly, a critical analysis of the works collected is conducted by identifying strengths and research trends in this domain.

Terrestrial laser scanning: a new standard of forest measuring and modelling?

BACKGROUND

Laser scanning technology has opened new horizons for the research of forest dynamics, because it provides a largely automated and non-destructive method to rapidly capture the structure of individual trees and entire forest stands at multiple spatial scales. The structural data themselves or in combination with additional remotely sensed data also provide information on the local physiological state of structures within trees. The capacity of new methods is facilitated by the ongoing development of automated processing tools that are designed to capture information from the point cloud data provided by the remote measurements.

SCOPE

Terrestrial laser scanning (TLS), performed from the ground or from unmanned aerial vehicles, in particular, has potential to become a unifying measurement standard for forest research questions, because the equipment is flexible to use in the field and has the capacity to capture branch-level structural information at the forestplot or even forest scale. This issue of Annals of Botany includes selected papers that exemplify the current and potential uses of TLS, such as for examination of crown interactions between trees, growth dynamics of mixed stands, non-destructive characterization of urban trees, and enhancement of ecological and evolutionary models. The papers also present current challenges in the applicability of TLS methods and report recent developments in methods facilitating the use of TLS data for research purposes, including automatic processing chains and quantifying branch and above-ground biomass. In this article, we provide an overview of the current and anticipated future capacity of TLS and related methods in solving questions that utilize measurements and models of forests.

CONCLUSIONS

Due to its measurement speed, TLS provides a method to effortlessly capture large amounts of detailed structural forest information, and consequent proxy data for tree and forest processes, at a far wider spatial scale than is feasible with manual measurements. Issues with measurement precision and occlusion of laser beams before they reach their target structures continue to reduce the accuracy of TLS data, but the limitations are counterweighted by the measurement speed that enables large sample sizes. The currently high time-cost of analysing TLS data, in turn, is likely to decrease through progress in automated processing methods. The developments point towards TLS becoming a new and widely accessible standard tool in forest measurement and modelling.

Visible and Thermal Image-Based Trunk Detection with Deep Learning for Forestry Mobile Robotics

Mobile robotics in forests is currently a hugely important topic due to the recurring appearance of forest wildfires. Thus, in-site management of forest inventory and biomass is required. To tackle this issue, this work presents a study on detection at the ground level of forest tree trunks in visible and thermal images using deep learning-based object detection methods. For this purpose, a forestry dataset composed of 2895 images was built and made publicly available. Using this dataset, five models were trained and benchmarked to detect the tree trunks. The selected models were SSD MobileNetV2, SSD Inception-v2, SSD ResNet50, SSDLite MobileDet and YOLOv4 Tiny. Promising results were obtained; for instance, YOLOv4 Tiny was the best model that achieved the highest AP (90%) and F1 score (89%). The inference time was also evaluated, for these models, on CPU and GPU. The results showed that YOLOv4 Tiny was the fastest detector running on GPU (8 ms). This work will enhance the development of vision perception systems for smarter forestry robots.

Deep Learning in Forestry Using UAV-Acquired RGB Data: A Practical Review

Forests are the planet’s main CO2 filtering agent as well as important economical, environmental and social assets. Climate change is exerting an increased stress, resulting in a need for improved research methodologies to study their health, composition or evolution. Traditionally, information about forests has been collected using expensive and work-intensive field inventories, but in recent years unoccupied autonomous vehicles (UAVs) have become very popular as they represent a simple and inexpensive way to gather high resolution data of large forested areas. In addition to this trend, deep learning (DL) has also been gaining much attention in the field of forestry as a way to include the knowledge of forestry experts into automatic software pipelines tackling problems such as tree detection or tree health/species classification. Among the many sensors that UAVs can carry, RGB cameras are fast, cost-effective and allow for straightforward data interpretation. This has resulted in a large increase in the amount of UAV-acquired RGB data available for forest studies. In this review, we focus on studies that use DL and RGB images gathered by UAVs to solve practical forestry research problems. We summarize the existing studies, provide a detailed analysis of their strengths paired with a critical assessment on common methodological problems and include other information, such as available public data and code resources that we believe can be useful for researchers that want to start working in this area. We structure our discussion using three main families of forestry problems: (1) individual Tree Detection, (2) tree Species Classification, and (3) forest Anomaly Detection (forest fires and insect Infestation).

Recent Advances in Unmanned Aerial Vehicles Forest Remote Sensing—A Systematic Review. Part II: Research Applications

Forest sustainable management aims to maintain the income of woody goods for companies, together with preserving non-productive functions as a benefit for the community. Due to the progress in platforms and sensors and the opening of the dedicated market, unmanned aerial vehicle–remote sensing (UAV–RS) is improving its key role in the forestry sector as a tool for sustainable management. The use of UAV (Unmanned Aerial Vehicle) in precision forestry has exponentially increased in recent years, as demonstrated by more than 600 references published from 2018 until mid-2020 that were found in the Web of Science database by searching for “UAV” + “forest”. This result is even more surprising when compared with similar research for “UAV” + “agriculture”, from which emerge about 470 references. This shows how UAV–RS research forestry is gaining increasing popularity. In Part II of this review, analyzing the main findings of the reviewed papers (227), numerous strengths emerge concerning research technical issues. UAV–RS is fully applicated for obtaining accurate information from practical parameters (height, diameter at breast height (DBH), and biomass). Research effectiveness and soundness demonstrate that UAV–RS is now ready to be applied in a real management context. Some critical issues and barriers in transferring research products are also evident, namely, (1) hyperspectral sensors are poorly used, and their novel applications should be based on the capability of acquiring tree spectral signature especially for pest and diseases detection, (2) automatic processes for image analysis are poorly flexible or based on proprietary software at the expense of flexible and open-source tools that can foster researcher activities and support technology transfer among all forestry stakeholders, and (3) a clear lack exist in sensors and platforms interoperability for large-scale applications and for enabling data interoperability.

Advances in Forest Robotics: A State-of-the-Art Survey

The development of robotic systems to operate in forest environments is of great relevance for the public and private sectors. In this sense, this article reviews several scientific papers, research projects and commercial products related to robotic applications for environmental preservation, monitoring, wildfire firefighting, inventory operations, planting, pruning and harvesting. After conducting critical analysis, the main characteristics observed were: (a) the locomotion system is directly affected by the type of environmental monitoring to be performed; (b) different reasons for pruning result in different locomotion and cutting systems; (c) each type of forest, in each season and each type of soil can directly interfere with the navigation technique used; and (d) the integration of the concept of swarm of robots with robots of different types of locomotion systems (land, air or sea) can compensate for the time of executing tasks in unstructured environments. Two major areas are proposed for future research works: Internet of Things (IoT)-based smart forest and navigation systems. It is expected that, with the various characteristics exposed in this paper, the current robotic forest systems will be improved, so that forest exploitation becomes more efficient and sustainable.

Recent Advances in Unmanned Aerial Vehicle Forest Remote Sensing—A Systematic Review. Part I: A General Framework

Natural, semi-natural, and planted forests are a key asset worldwide, providing a broad range of positive externalities. For sustainable forest planning and management, remote sensing (RS) platforms are rapidly going mainstream. In a framework where scientific production is growing exponentially, a systematic analysis of unmanned aerial vehicle (UAV)-based forestry research papers is of paramount importance to understand trends, overlaps and gaps. The present review is organized into two parts (Part I and Part II). Part II inspects specific technical issues regarding the application of UAV-RS in forestry, together with the pros and cons of different UAV solutions and activities where additional effort is needed, such as the technology transfer. Part I systematically analyzes and discusses general aspects of applying UAV in natural, semi-natural and artificial forestry ecosystems in the recent peer-reviewed literature (2018–mid-2020). The specific goals are threefold: (i) create a carefully selected bibliographic dataset that other researchers can draw on for their scientific works; (ii) analyze general and recent trends in RS forest monitoring (iii) reveal gaps in the general research framework where an additional activity is needed. Through double-step filtering of research items found in the Web of Science search engine, the study gathers and analyzes a comprehensive dataset (226 articles). Papers have been categorized into six main topics, and the relevant information has been subsequently extracted. The strong points emerging from this study concern the wide range of topics in the forestry sector and in particular the retrieval of tree inventory parameters often through Digital Aerial Photogrammetry (DAP), RGB sensors, and machine learning techniques. Nevertheless, challenges still exist regarding the promotion of UAV-RS in specific parts of the world, mostly in the tropical and equatorial forests. Much additional research is required for the full exploitation of hyperspectral sensors and for planning long-term monitoring.

Canopy Top, Height and Photosynthetic Pigment Estimation Using Parrot Sequoia Multispectral Imagery and the Unmanned Aerial Vehicle (UAV)

Remote sensing is one of the modern methods that have significantly developed over the last two decades and, nowadays, it provides a new means for forest monitoring. High spatial and temporal resolutions are demanded for the accurate and timely monitoring of forests. In this study, multi-spectral Unmanned Aerial Vehicle (UAV) images were used to estimate canopy parameters (definition of crown extent, top, and height, as well as photosynthetic pigment contents). The UAV images in Green, Red, Red-Edge, and Near infrared (NIR) bands were acquired by Parrot Sequoia camera over selected sites in two small catchments (Czech Republic) covered dominantly by Norway spruce monocultures. Individual tree extents, together with tree tops and heights, were derived from the Canopy Height Model (CHM). In addition, the following were tested: (i) to what extent can the linear relationship be established between selected vegetation indexes (Normalized Difference Vegetation Index (NDVI) and NDVIred edge) derived for individual trees and the corresponding ground truth (e.g., biochemically assessed needle photosynthetic pigment contents) and (ii) whether needle age selection as a ground truth and crown light conditions affect the validity of linear models. The results of the conducted statistical analysis show that the two vegetation indexes (NDVI and NDVIred edge) tested here have the potential to assess photosynthetic pigments in Norway spruce forests at a semi-quantitative level; however, the needle-age selection as a ground truth was revealed to be a very important factor. The only usable results were obtained for linear models when using the second year needle pigment contents as a ground truth. On the other hand, the illumination conditions of the crown proved to have very little effect on the model’s validity. No study was found to directly compare these results conducted on coniferous forest stands. This shows that there is a further need for studies dealing with a quantitative estimation of the biochemical variables of nature coniferous forests when employing spectral data that were acquired by the UAV platform at a very high spatial resolution.

Effects of Differences in Structure from Motion Software on Image Processing of Unmanned Aerial Vehicle Photography and Estimation of Crown Area and Tree Height in Forests

This study examines the effects of differences in structure from motion (SfM) software on image processing of aerial images by unmanned aerial vehicles (UAV) and the resulting estimations of tree height and tree crown area. There were 20 flight conditions for the UAV aerial images, which were a combination of five conditions for flight altitude, two conditions for overlap, and two conditions for side overlap. Images were then processed using three SfM programs (Terra Mapper, PhotoScan, and Pix4Dmapper). The tree height and tree crown area were determined, and the SfM programs were compared based on the estimations. The number of densified point clouds for PhotoScan (160 × 105 to 50 × 105) was large compared to the two other two SfM programs. The estimated values of crown area and tree height by each SfM were compared via Bonferroni multiple comparisons (statistical significance level set at p < 0.05). The estimated values of canopy area showed statistically significant differences (p < 0.05) in 14 flight conditions for Terra Mapper and PhotoScan, 16 flight conditions for Terra Mapper and Pix4Dmapper, and 11 flight conditions for PhotoScan and Pix4Dmappers. In addition, the estimated values of tree height showed statistically significant differences (p < 0.05) in 15 flight conditions for Terra Mapper and PhotoScan, 19 flight conditions for Terra Mapper and Pix4Dmapper, and 20 flight conditions for PhotoScan and Pix4Dmapper. The statistically significant difference (p < 0.05) between the estimated value and measured value of each SfM was confirmed under 18 conditions for Terra Mapper, 20 conditions for PhotoScan, and 13 conditions for Pix4D. Moreover, the RMSE and rRMSE values of the estimated tree height were 5–6 m and 20–28%, respectively. Although the estimation accuracy of any SfM was low, the estimated tree height by Pix4D in many flight conditions had smaller RMSE values than the other software. As statistically significant differences were found between the SfMs in many flight conditions, we conclude that there were differences in the estimates of crown area and tree height depending on the SfM used. In addition, Pix4Dmapper is suitable for estimating forest information, such as tree height, and PhotoScan is suitable for detailed monitoring of disaster areas.

Field-based individual plant phenotyping of herbaceous species by unmanned aerial vehicle

Recent advances in Unmanned Aerial Vehicle (UAVs) and image processing have made high-throughput field phenotyping possible at plot/canopy level in the mass grown experiment. Such techniques are now expected to be used for individual level phenotyping in the single grown experiment.We found two main challenges of phenotyping individual plants in the single grown experiment: plant segmentation from weedy backgrounds and the estimation of complex traits that are difficult to measure manually.In this study, we proposed a methodological framework for field-based individual plant phenotyping by UAV. Two contributions, which are weed elimination for individual plant segmentation, and complex traits (volume and outline) extraction, have been developed. The framework demonstrated its utility in the phenotyping of Helianthus tuberosus (Jerusalem artichoke), an herbaceous perennial plant species.The proposed framework can be applied to either small and large scale phenotyping experiments.

A Review on Drone-Based Data Solutions for Cereal Crops

Food security is a longstanding global issue over the last few centuries. Eradicating hunger and all forms of malnutrition by 2030 is still a key challenge. The COVID-19 pandemic has placed additional stress on food production, demand, and supply chain systems; majorly impacting cereal crop producer and importer countries. Short food supply chain based on the production from local farms is less susceptible to travel and export bans and works as a smooth system in the face of these stresses. Local drone-based data solutions can provide an opportunity to address these challenges. This review aims to present a deeper understanding of how the drone-based data solutions can help to combat food insecurity caused due to the pandemic, zoonotic diseases, and other food shocks by enhancing cereal crop productivity of small-scale farming systems in low-income countries. More specifically, the review covers sensing capabilities, promising algorithms, and methods, and added-value of novel machine learning algorithms for local-scale monitoring, biomass and yield estimation, and mapping of them. Finally, we present the opportunities for linking information from citizen science, internet of things (IoT) based on low-cost sensors and drone-based information to satellite data for upscaling crop yield estimation to a larger geographical extent within the Earth Observation umbrella.

A Transferable and Effective Method for Monitoring Continuous Cover Forestry at the Individual Tree Level Using UAVs

Transformation to Continuous Cover Forestry (CCF) is a long and difficult process in which frequent management interventions rapidly alter forest structure and dynamics with long lasting impacts. Therefore, a critical component of transformation is the acquisition of up-to-date forest inventory data to direct future management decisions. Recently, the use of single tree detection methods derived from unmanned aerial vehicle (UAV) has been identified as being a cost effective method for inventorying forests. However, the rapidly changing structure of forest stands in transformation amplifies the difficultly in transferability of current individual tree detection (ITD) methods. This study presents a novel ITD Bayesian parameter optimisation approach that uses quantile regression and external biophysical tree data sets to provide a transferable and low cost ITD approach to monitoring stands in transformation. We applied this novel method to 5 stands in a variety of transformation stages in the UK and to a independent test study site in California, USA, to assess the accuracy and transferability of this method. Requiring small amounts of training data (15 reference trees) this approach had a mean test accuracy (F-score = 0.88) and provided mean tree diameter estimates (RMSE = 5.6 cm) with differences that were not significance to the ground data (p < 0.05). We conclude that this method can be used to monitor forests stands in transformation and thus can also be applied to a wide range of forest structures with limited manual parameterisation between sites.

An Automatic UAV Based Segmentation Approach for Pruning Biomass Estimation in Irregularly Spaced Chestnut Orchards

The agricultural and forestry sector is constantly evolving, also through the increased use of precision technologies including Remote Sensing (RS). Remotely biomass estimation (WaSfM) in wood production forests is already debated in the literature, but there is a lack of knowledge in quantifying pruning residues from canopy management. The aim of the present study was to verify the reliability of RS techniques for the estimation of pruning biomass through differences in the volume of canopy trees and to evaluate the performance of an unsupervised segmentation methodology as a feasible tool for the analysis of large areas. Remote sensed data were acquired on four uneven-aged and irregularly spaced chestnut orchards in Central Italy by an Unmanned Aerial Vehicle (UAV) equipped with a multispectral camera. Chestnut geometric features were extracted using both supervised and unsupervised crown segmentation and then applying a double filtering process based on Canopy Height Model (CHM) and vegetation index threshold. The results show that UAV monitoring provides good performance in detecting biomass reduction after pruning, despite some differences between the trees’ geometric features. The proposed unsupervised methodology for tree detection and vegetation cover evaluation purposes showed good performance, with a low undetected tree percentage value (1.7%). Comparing crown projected volume reduction extracted by means of supervised and unsupervised approach, R2 ranged from 0.76 to 0.95 among all the sites. Finally, the validation step was assessed by evaluating correlations between measured and estimated pruning wood biomass (Wpw) for single and grouped sites (0.53 < R2 < 0.83). The method described in this work could provide effective strategic support for chestnut orchard management in line with a precision agriculture approach. In the context of the Circular Economy, a fast and cost-effective tool able to estimate the amounts of wastes available as by-products such as chestnut pruning residues can be included in an alternative and virtuous supply chain.