In-Field High-Throughput Phenotyping of Cotton Plant Height Using LiDAR

Calibrating ultrasonic sensor measurements of crop canopy heights: a case study of maize and wheat

Canopy height serves as an important dynamic indicator of crop growth in the decision-making process of field management. Compared with other commonly used canopy height measurement techniques, ultrasonic sensors are inexpensive and can be exposed in fields for long periods of time to obtain easy-to-process data. However, the acoustic wave characteristics and crop canopy structure affect the measurement accuracy. To improve the ultrasonic sensor measurement accuracy, a four-year (2018-2021) field experiment was conducted on maize and wheat, and a measurement platform was developed. A series of single-factor experiments were conducted to investigate the significant factors affecting measurements, including the observation angle (0-60°), observation height (0.5-2.5 m), observation period (8:00-18:00), platform moving speed with respect to the crop (0-2.0 m min-1), planting density (0.2-1 time of standard planting density), and growth stage (maize from three-leaf to harvest period and wheat from regreening to maturity period). The results indicated that both the observation angle and planting density significantly affected the results of ultrasonic measurements (p-value< 0.05), whereas the effects of other factors on measurement accuracy were negligible (p-value > 0.05). Moreover, a double-input factor calibration model was constructed to assess canopy height under different years by utilizing the normalized difference vegetation index and ultrasonic measurements. The model was developed by employing the least-squares method, and ultrasonic measurement accuracy was significantly improved when integrating the measured value of canopy heights and the normalized difference vegetation index (NDVI). The maize measurement accuracy had a root mean squared error (RMSE) ranging from 81.4 mm to 93.6 mm, while the wheat measurement accuracy had an RMSE from 37.1 mm to 47.2 mm. The research results effectively combine stable and low-cost commercial sensors with ground-based agricultural machinery platforms, enabling efficient and non-destructive acquisition of crop height information.

HairNet2: deep learning to quantify cotton leaf hairiness, a complex genetic and environmental trait

BACKGROUND

Cotton accounts for 80% of the global natural fibre production. Its leaf hairiness affects insect resistance, fibre yield, and economic value. However, this phenotype is still qualitatively assessed by visually attributing a Genotype Hairiness Score (GHS) to a leaf/plant, or by using the HairNet deep-learning model which also outputs a GHS. Here, we introduce HairNet2, a quantitative deep-learning model which detects leaf hairs (trichomes) from images and outputs a segmentation mask and a Leaf Trichome Score (LTS).

RESULTS

Trichomes of 1250 images were annotated (AnnCoT) and a combination of six Feature Extractor modules and five Segmentation modules were tested alongside a range of loss functions and data augmentation techniques. HairNet2 was further validated on the dataset used to build HairNet (CotLeaf-1), a similar dataset collected in two subsequent seasons (CotLeaf-2), and a dataset collected on two genetically diverse populations (CotLeaf-X). The main findings of this study are that (1) leaf number, environment and image position did not significantly affect results, (2) although GHS and LTS mostly correlated for individual GHS classes, results at the genotype level revealed a strong LTS heterogeneity within a given GHS class, (3) LTS correlated strongly with expert scoring of individual images.

CONCLUSIONS

HairNet2 is the first quantitative and scalable deep-learning model able to measure leaf hairiness. Results obtained with HairNet2 concur with the qualitative values used by breeders at both extremes of the scale (GHS 1-2, and 5-5+), but interestingly suggest a reordering of genotypes with intermediate values (GHS 3-4+). Finely ranking mild phenotypes is a difficult task for humans. In addition to providing assistance with this task, HairNet2 opens the door to selecting plants with specific leaf hairiness characteristics which may be associated with other beneficial traits to deliver better varieties.

A comparative study on point cloud down-sampling strategies for deep learning-based crop organ segmentation

The 3D crop data obtained during cultivation is of great significance to screening excellent varieties in modern breeding and improvement on crop yield. With the rapid development of deep learning, researchers have been making innovations in aspects of both data preparation and deep network design for segmenting plant organs from 3D data. Training of the deep learning network requires the input point cloud to have a fixed scale, which means all point clouds in the batch should have similar scale and contain the same number of points. A good down-sampling strategy can reduce the impact of noise and meanwhile preserve the most important 3D spatial structures. As far as we know, this work is the first comprehensive study of the relationship between multiple down-sampling strategies and the performances of popular networks for plant point clouds. Five down-sampling strategies (including FPS, RS, UVS, VFPS, and 3DEPS) are cross evaluated on five different segmentation networks (including PointNet +  + , DGCNN, PlantNet, ASIS, and PSegNet). The overall experimental results show that currently there is no strict golden rule on fixing down-sampling strategy for a specific mainstream crop deep learning network, and the optimal down-sampling strategy may vary on different networks. However, some general experience for choosing an appropriate sampling method for a specific network can still be summarized from the qualitative and quantitative experiments. First, 3DEPS and UVS are easy to generate better results on semantic segmentation networks. Second, the voxel-based down-sampling strategies may be more suitable for complex dual-function networks. Third, at 4096-point resolution, 3DEPS usually has only a small margin compared with the best down-sampling strategy at most cases, which means 3DEPS may be the most stable strategy across all compared. This study not only helps to further improve the accuracy of point cloud deep learning networks for crop organ segmentation, but also gives clue to the alignment of down-sampling strategies and a specific network.

Field-measured canopy height may not be as accurate and heritable as believed: evidence from advanced 3D sensing

Canopy height (CH) is an important trait for crop breeding and production. The rapid development of 3D sensing technologies shed new light on high-throughput height measurement. However, a systematic comparison of the accuracy and heritability of different 3D sensing technologies is seriously lacking. Moreover, it is questionable whether the field-measured height is as reliable as believed. This study uncovered these issues by comparing traditional height measurement with four advanced 3D sensing technologies, including terrestrial laser scanning (TLS), backpack laser scanning (BLS), gantry laser scanning (GLS), and digital aerial photogrammetry (DAP). A total of 1920 plots covering 120 varieties were selected for comparison. Cross-comparisons of different data sources were performed to evaluate their performances in CH estimation concerning different CH, leaf area index (LAI), and growth stage (GS) groups. Results showed that 1) All 3D sensing data sources had high correlations with field measurement (r > 0.82), while the correlations between different 3D sensing data sources were even better (r > 0.87). 2) The prediction accuracy between different data sources decreased in subgroups of CH, LAI, and GS. 3) Canopy height showed high heritability from all datasets, and 3D sensing datasets had even higher heritability (H² = 0.79-0.89) than FM (field measurement) (H² = 0.77). Finally, outliers of different datasets are analyzed. The results provide novel insights into different methods for canopy height measurement that may ensure the high-quality application of this important trait.

Proximal and remote sensing in plant phenomics: 20 years of progress, challenges, and perspectives

Plant phenomics (PP) has been recognized as a bottleneck in studying the interactions of genomics and environment on plants, limiting the progress of smart breeding and precise cultivation. High-throughput plant phenotyping is challenging owing to the spatio-temporal dynamics of traits. Proximal and remote sensing (PRS) techniques are increasingly used for plant phenotyping because of their advantages in multi-dimensional data acquisition and analysis. Substantial progress of PRS applications in PP has been observed over the last two decades and is analyzed here from an interdisciplinary perspective based on 2972 publications. This progress covers most aspects of PRS application in PP, including patterns of global spatial distribution and temporal dynamics, specific PRS technologies, phenotypic research fields, working environments, species, and traits. Subsequently, we demonstrate how to link PRS to multi-omics studies, including how to achieve multi-dimensional PRS data acquisition and processing, how to systematically integrate all kinds of phenotypic information and derive phenotypic knowledge with biological significance, and how to link PP to multi-omics association analysis. Finally, we identify three future perspectives for PRS-based PP: (1) strengthening the spatial and temporal consistency of PRS data, (2) exploring novel phenotypic traits, and (3) facilitating multi-omics communication.

UAV Image-Based Crop Growth Analysis of 3D-Reconstructed Crop Canopies

Plant growth rate is an essential phenotypic parameter for quantifying potential crop productivity. Under field conditions, manual measurement of plant growth rate is less accurate in most cases. Image-based high-throughput platforms offer great potential for rapid, non-destructive, and objective estimation of plant growth parameters. The aim of this study was to assess the potential for quantifying plant growth rate using UAV-based (unoccupied aerial vehicle) imagery collected multiple times throughout the growing season. In this study, six diverse lines of lentils were grown in three replicates of 1 m² microplots with six biomass collection time-points throughout the growing season over five site-years. Aerial imagery was collected simultaneously with each manual measurement of the above-ground biomass time-point and was used to produce two-dimensional orthomosaics and three-dimensional point clouds. Non-linear logistic models were fit to multiple data collection points throughout the growing season. Overall, remotely detected vegetation area and crop volume were found to produce trends comparable to the accumulation of dry weight biomass throughout the growing season. The growth rate and G50 (days to 50% of maximum growth) parameters of the model effectively quantified lentil growth rate indicating significant potential for image-based tools to be used in plant breeding programs. Comparing image-based groundcover and vegetation volume estimates with manually measured above-ground biomass suggested strong correlations. Vegetation area measured from a UAV has utility in quantifying lentil biomass and is indicative of leaf area early in the growing season. For mid- to late-season biomass estimation, plot volume was determined to be a better estimator. Apart from traditional traits, the estimation and analysis of plant parameters not typically collected in traditional breeding programs are possible with image-based methods, and this can create new opportunities to improve breeding efficiency mainly by offering new phenotypes and affecting selection intensity.

The field phenotyping platform's next darling: Dicotyledons

The genetic information and functional properties of plants have been further identified with the completion of the whole-genome sequencing of numerous crop species and the rapid development of high-throughput phenotyping technologies, laying a suitable foundation for advanced precision agriculture and enhanced genetic gains. Collecting phenotypic data from dicotyledonous crops in the field has been identified as a key factor in the collection of large-scale phenotypic data of crops. On the one hand, dicotyledonous plants account for 4/5 of all angiosperm species and play a critical role in agriculture. However, their morphology is complex, and an abundance of dicot phenotypic information is available, which is critical for the analysis of high-throughput phenotypic data in the field. As a result, the focus of this paper is on the major advancements in ground-based, air-based, and space-based field phenotyping platforms over the last few decades and the research progress in the high-throughput phenotyping of dicotyledonous field crop plants in terms of morphological indicators, physiological and biochemical indicators, biotic/abiotic stress indicators, and yield indicators. Finally, the future development of dicots in the field is explored from the perspectives of identifying new unified phenotypic criteria, developing a high-performance infrastructure platform, creating a phenotypic big data knowledge map, and merging the data with those of multiomic techniques.

A Deep Learning-Based System for Monitoring the Number and Height Growth Rates of Moso Bamboo Shoots

The number and growth of new shoots are very important information for bamboo forest cultivation and management. At present, there is no real-time, efficient and accurate monitoring method. In this study, a fixed webcam was applied for image capture, optimized YOLOv4 was used to model the detection of moso bamboo shoots, and a strategy of sorting and screening was proposed to track each moso bamboo shoot. The change in the number and height of moso bamboo shoots was obtained according to the number and height of detection boxes. The experimental results show that the system can remotely and automatically obtain the number of moso bamboo shoots and the pixel height of each bamboo shoot at any given time. The average relative error and variance in the number of moso bamboo shoots were 1.28% and 0.016%, respectively, and those for the corresponding pixel height results were −0.39% and 0.02%. This system can be applied to a series of monitoring purposes, such as the daily or weekly growth rate of moso bamboo shoots at monitoring stations and trends in the height of selected bamboo shoots.

New strategies on the application of artificial intelligence in the field of phytoremediation

Artificial Intelligence (AI) is expected to play a crucial role in the field of phytoremediation and its effective management in monitoring the growth of the plant in different contaminated soils and their phenotype characteristic such as the biomass of plants. This review focuses on recent applications of various AI techniques and remote sensing approaches in the field of phytoremediation to monitor plant growth with relevant morphological parameters using novel sensors, cameras, and associated modern technologies. Novel sensing and various measurement techniques are highlighted. Input parameters are used to develop futuristic models utilizing AI and statistical approaches. Additionally, a brief discussion has been presented on the use of AI techniques to detect metal hyperaccumulation in all parts of the plant, carbon capture, and sequestration along with its effect on food production to ensure food safety and security. This article highlights the application, limitation, and future perspectives of phytoremediation in monitoring the mobility, bioavailability, seasonal variation, effect of temperature on plant growth, and plant response to the heavy metals in soil by using the AI technique. Suggestions are made for future research in this area to analyze which would help to enhance plant growth and improve food security in long run.

PSegNet: Simultaneous Semantic and Instance Segmentation for Point Clouds of Plants

Phenotyping of plant growth improves the understanding of complex genetic traits and eventually expedites the development of modern breeding and intelligent agriculture. In phenotyping, segmentation of 3D point clouds of plant organs such as leaves and stems contributes to automatic growth monitoring and reflects the extent of stress received by the plant. In this work, we first proposed the Voxelized Farthest Point Sampling (VFPS), a novel point cloud downsampling strategy, to prepare our plant dataset for training of deep neural networks. Then, a deep learning network—PSegNet, was specially designed for segmenting point clouds of several species of plants. The effectiveness of PSegNet originates from three new modules including the Double-Neighborhood Feature Extraction Block (DNFEB), the Double-Granularity Feature Fusion Module (DGFFM), and the Attention Module (AM). After training on the plant dataset prepared with VFPS, the network can simultaneously realize the semantic segmentation and the leaf instance segmentation for three plant species. Comparing to several mainstream networks such as PointNet++, ASIS, SGPN, and PlantNet, the PSegNet obtained the best segmentation results quantitatively and qualitatively. In semantic segmentation, PSegNet achieved 95.23%, 93.85%, 94.52%, and 89.90% for the mean Prec, Rec, F1, and IoU, respectively. In instance segmentation, PSegNet achieved 88.13%, 79.28%, 83.35%, and 89.54% for the mPrec, mRec, mCov, and mWCov, respectively.

CBM: An IoT Enabled LiDAR Sensor for In-Field Crop Height and Biomass Measurements

The phenotypic characterization of crop genotypes is an essential, yet challenging, aspect of crop management and agriculture research. Digital sensing technologies are rapidly advancing plant phenotyping and speeding-up crop breeding outcomes. However, off-the-shelf sensors might not be fully applicable and suitable for agricultural research due to the diversity in crop species and specific needs during plant breeding selections. Customized sensing systems with specialized sensor hardware and software architecture provide a powerful and low-cost solution. This study designed and developed a fully integrated Raspberry Pi-based LiDAR sensor named CropBioMass (CBM), enabled by internet of things to provide a complete end-to-end pipeline. The CBM is a low-cost sensor, provides high-throughput seamless data collection in field, small data footprint, injection of data onto the remote server, and automated data processing. The phenotypic traits of crop fresh biomass, dry biomass, and plant height that were estimated by CBM data had high correlation with ground truth manual measurements in a wheat field trial. The CBM is readily applicable for high-throughput plant phenotyping, crop monitoring, and management for precision agricultural applications.

Exploring Seasonal and Circadian Rhythms in Structural Traits of Field Maize from LiDAR Time Series

Plant growth rhythm in structural traits is important for better understanding plant response to the ever-changing environment. Terrestrial laser scanning (TLS) is a well-suited tool to study structural rhythm under field conditions. Recent studies have used TLS to describe the structural rhythm of trees, but no consistent patterns have been drawn. Meanwhile, whether TLS can capture structural rhythm in crops is unclear. Here, we aim to explore the seasonal and circadian rhythms in maize structural traits at both the plant and leaf levels from time-series TLS. The seasonal rhythm was studied using TLS data collected at four key growth periods, including jointing, bell-mouthed, heading, and maturity periods. Circadian rhythms were explored by using TLS data acquired around every 2 hours in a whole day under standard and cold stress conditions. Results showed that TLS can quantify the seasonal and circadian rhythm in structural traits at both plant and leaf levels. (1) Leaf inclination angle decreased significantly between the jointing stage and bell-mouthed stage. Leaf azimuth was stable after the jointing stage. (2) Some individual-level structural rhythms (e.g., azimuth and projected leaf area/PLA) were consistent with leaf-level structural rhythms. (3) The circadian rhythms of some traits (e.g., PLA) were not consistent under standard and cold stress conditions. (4) Environmental factors showed better correlations with leaf traits under cold stress than standard conditions. Temperature was the most important factor that significantly correlated with all leaf traits except leaf azimuth. This study highlights the potential of time-series TLS in studying outdoor agricultural chronobiology.

Three-Dimensional Reconstruction Method of Rapeseed Plants in the Whole Growth Period Using RGB-D Camera

The three-dimensional reconstruction method using RGB-D camera has a good balance in hardware cost and point cloud quality. However, due to the limitation of inherent structure and imaging principle, the acquired point cloud has problems such as a lot of noise and difficult registration. This paper proposes a 3D reconstruction method using Azure Kinect to solve these inherent problems. Shoot color images, depth images and near-infrared images of the target from six perspectives by Azure Kinect sensor with black background. Multiply the binarization result of the 8-bit infrared image with the RGB-D image alignment result provided by Microsoft corporation, which can remove ghosting and most of the background noise. A neighborhood extreme filtering method is proposed to filter out the abrupt points in the depth image, by which the floating noise point and most of the outlier noise will be removed before generating the point cloud, and then using the pass-through filter eliminate rest of the outlier noise. An improved method based on the classic iterative closest point (ICP) algorithm is presented to merge multiple-views point clouds. By continuously reducing both the size of the down-sampling grid and the distance threshold between the corresponding points, the point clouds of each view are continuously registered three times, until get the integral color point cloud. Many experiments on rapeseed plants show that the success rate of cloud registration is 92.5% and the point cloud accuracy obtained by this method is 0.789 mm, the time consuming of a integral scanning is 302 s, and with a good color restoration. Compared with a laser scanner, the proposed method has considerable reconstruction accuracy and a significantly ahead of the reconstruction speed, but the hardware cost is much lower when building a automatic scanning system. This research shows a low-cost, high-precision 3D reconstruction technology, which has the potential to be widely used for non-destructive measurement of rapeseed and other crops phenotype.

Opportunities for Robotic Systems and Automation in Cotton Production

Automation continues to play a greater role in agricultural production with commercial systems now available for machine vision identification of weeds and other pests, autonomous weed control, and robotic harvesters for fruits and vegetables. The growing availability of autonomous machines in agriculture indicates that there are opportunities to increase automation in cotton production. This article considers how current and future advances in automation has, could, or will impact cotton production practices. The results are organized to follow the cotton production process from land preparation to planting to within season management through harvesting and ginning. For each step, current and potential opportunities to automate processes are discussed. Specific examples include advances in automated weed control and progress made in the use of robotic systems for cotton harvesting.

Optimization of 3D Point Clouds of Oilseed Rape Plants Based on Time-of-Flight Cameras

Three-dimensional (3D) structure is an important morphological trait of plants for describing their growth and biotic/abiotic stress responses. Various methods have been developed for obtaining 3D plant data, but the data quality and equipment costs are the main factors limiting their development. Here, we propose a method to improve the quality of 3D plant data using the time-of-flight (TOF) camera Kinect V2. A K-dimension (k-d) tree was applied to spatial topological relationships for searching points. Background noise points were then removed with a minimum oriented bounding box (MOBB) with a pass-through filter, while outliers and flying pixel points were removed based on viewpoints and surface normals. After being smoothed with the bilateral filter, the 3D plant data were registered and meshed. We adjusted the mesh patches to eliminate layered points. The results showed that the patches were closer. The average distance between the patches was 1.88 × 10-3 m, and the average angle was 17.64°, which were 54.97% and 48.33% of those values before optimization. The proposed method performed better in reducing noise and the local layered-points phenomenon, and it could help to more accurately determine 3D structure parameters from point clouds and mesh models.

An automatic method for counting wheat tiller number in the field with terrestrial LiDAR

BACKGROUND

The tiller number per unit area is one of the main agronomic components in determining yield. A real-time assessment of this trait could contribute to monitoring the growth of wheat populations or as a primary phenotyping indicator for the screening of cultivars for crop breeding. However, determining tiller number has been conventionally dependent on tedious and labor-intensive manual counting. In this study, an automatic tiller-counting algorithm was developed to estimate the tiller density under field conditions based on terrestrial laser scanning (TLS) data. The novel algorithm, which is named ALHC, involves two steps: (1) the use of an adaptive layering (AL) algorithm for cluster segmentation and (2) the use of a hierarchical clustering (HC) algorithm for tiller detection among the clusters. Three field trials during the 2016-2018 wheat seasons were conducted to validate the algorithm with twenty different wheat cultivars, three nitrogen levels, and two planting densities at two ecological sites (Rugao & Xuzhou) in Jiangsu Province, China.

RESULT

The results demonstrated that the algorithm was promising across different cultivars, years, growth stages, planting densities, and ecological sites. The tests from Rugao and Xuzhou in 2016-2017 and Rugao in 2017-2018 showed that the algorithm estimated the tiller number of the wheat with regression coefficient (R²) values of 0.61, 0.56 and 0.65, respectively. In short, tiller counting with the ALHC generally underestimated the tiller number and performed better for the data with lower plant densities, compact plant types and the jointing stage, which were associated with overlap and noise between plants and inside the dense canopy.

CONCLUSIONS

Differing from the previous methods, the ALHC proposed in this paper made full use of 3D crop information and developed an automatic tiller counting method that is suitable for the field environment.

Quantification of light interception within image-based 3-D reconstruction of sole and intercropped canopies over the entire growth season

BACKGROUND AND AIMS

Light interception is closely related to canopy architecture. Few studies based on multi-view photography have been conducted in a field environment, particularly studies that link 3-D plant architecture with a radiation model to quantify the dynamic canopy light interception. In this study, we combined realistic 3-D plant architecture with a radiation model to quantify and evaluate the effect of differences in planting patterns and row orientations on canopy light interception.

METHODS

The 3-D architectures of maize and soybean plants were reconstructed for sole crops and intercrops based on multi-view images obtained at five growth dates in the field. We evaluated the accuracy of the calculated leaf length, maximum leaf width, plant height and leaf area according to the measured data. The light distribution within the 3-D plant canopy was calculated with a 3-D radiation model. Finally, we evaluated canopy light interception in different row orientations.

KEY RESULTS

There was good agreement between the measured and calculated phenotypic traits, with an R2 >0.97. The light distribution was more uniform for intercropped maize and more concentrated for sole maize. At the maize silking stage, 85 % of radiation was intercepted by approx. 55 % of the upper canopy region for maize and by approx. 33 % of the upper canopy region for soybean. There was no significant difference in daily light interception between the different row orientations for the entire intercropping and sole systems. However, for intercropped maize, near east-west orientations showed approx. 19 % higher daily light interception than near south-north orientations. For intercropped soybean, daily light interception showed the opposite trend. It was approx. 49 % higher for near south-north orientations than for near east-west orientations.

CONCLUSIONS

The accurate reconstruction of 3-D plants grown in the field based on multi-view images provides the possibility for high-throughput 3-D phenotyping in the field and allows a better understanding of the relationship between canopy architecture and the light environment.

Simulation of an Autonomous Mobile Robot for LiDAR-Based In-Field Phenotyping and Navigation

The agriculture industry is in need of substantially increasing crop yield to meet growing global demand. Selective breeding programs can accelerate crop improvement but collecting phenotyping data is time- and labor-intensive because of the size of the research fields and the frequency of the work required. Automation could be a promising tool to address this phenotyping bottleneck. This paper presents a Robotic Operating System (ROS)-based mobile field robot that simultaneously navigates through occluded crop rows and performs various phenotyping tasks, such as measuring plant volume and canopy height using a 2D LiDAR in a nodding configuration. The efficacy of the proposed 2D LiDAR configuration for phenotyping is assessed in a high-fidelity simulated agricultural environment in the Gazebo simulator with an ROS-based control framework and compared with standard LiDAR configurations used in agriculture. Using the proposed nodding LiDAR configuration, a strategy for navigation through occluded crop rows is presented. The proposed LiDAR configuration achieved an estimation error of 6.6% and 4% for plot volume and canopy height, respectively, which was comparable to the commonly used LiDAR configurations. The hybrid strategy with GPS waypoint following and LiDAR-based navigation was used to navigate the robot through an agricultural crop field successfully with an root mean squared error of 0.0778 m which was 0.2% of the total traveled distance. The presented robot simulation framework in ROS and optimized LiDAR configuration helped to expedite the development of the agricultural robots, which ultimately will aid in overcoming the phenotyping bottleneck.

An Extensive Review of Mobile Agricultural Robotics for Field Operations: Focus on Cotton Harvesting

In this review, we examine opportunities and challenges for 21st-century robotic agricultural cotton harvesting research and commercial development. The paper reviews opportunities present in the agricultural robotics industry, and a detailed analysis is conducted for the cotton harvesting robot industry. The review is divided into four sections: (1) general agricultural robotic operations, where we check the current robotic technologies in agriculture; (2) opportunities and advances in related robotic harvesting fields, which is focused on investigating robotic harvesting technologies; (3) status and progress in cotton harvesting robot research, which concentrates on the current research and technology development in cotton harvesting robots; and (4) challenges in commercial deployment of agricultural robots, where challenges to commercializing and using these robots are reviewed. Conclusions are drawn about cotton harvesting robot research and the potential of multipurpose robotic operations in general. The development of multipurpose robots that can do multiple operations on different crops to increase the value of the robots is discussed. In each of the sections except the conclusion, the analysis is divided into four robotic system categories; mobility and steering, sensing and localization, path planning, and robotic manipulation.

Biomass and Crop Height Estimation of Different Crops Using UAV-Based Lidar

Phenotyping of crops is important due to increasing pressure on food production. Therefore, an accurate estimation of biomass during the growing season can be important to optimize the yield. The potential of data acquisition by UAV-LiDAR to estimate fresh biomass and crop height was investigated for three different crops (potato, sugar beet, and winter wheat) grown in Wageningen (The Netherlands) from June to August 2018. Biomass was estimated using the 3DPI algorithm, while crop height was estimated using the mean height of a variable number of highest points for each m2. The 3DPI algorithm proved to estimate biomass well for sugar beet (R2 = 0.68, RMSE = 17.47 g/m2) and winter wheat (R2 = 0.82, RMSE = 13.94 g/m2). Also, the height estimates worked well for sugar beet (R2 = 0.70, RMSE = 7.4 cm) and wheat (R2 = 0.78, RMSE = 3.4 cm). However, for potato both plant height (R2 = 0.50, RMSE = 12 cm) and biomass estimation (R2 = 0.24, RMSE = 22.09 g/m2), it proved to be less reliable due to the complex canopy structure and the ridges on which potatoes are grown. In general, for accurate biomass and crop height estimates using those algorithms, the flight conditions (altitude, speed, location of flight lines) should be comparable to the settings for which the models are calibrated since changing conditions do influence the estimated biomass and crop height strongly.

Estimating Biomass and Canopy Height With LiDAR for Field Crop Breeding

Above-ground biomass (AGB) is a trait with much potential for exploitation within wheat breeding programs and is linked closely to canopy height (CH). However, collecting phenotypic data for AGB and CH within breeding programs is labor intensive, and in the case of AGB, destructive and prone to assessment error. As a result, measuring these traits is seldom a priority for breeders, especially at the early stages of a selection program. LiDAR has been demonstrated as a sensor capable of collecting three-dimensional data from wheat field trials, and potentially suitable for providing objective, non-destructive, high-throughput estimates of AGB and CH for use by wheat breeders. The current study investigates the deployment of a LiDAR system on a ground-based high-throughput phenotyping platform in eight wheat field trials across southern Australia, for the non-destructive estimate of AGB and CH. LiDAR-derived measurements were compared to manual measurements of AGB and CH collected at each site and assessed for their suitability of application within a breeding program. Correlations between AGB and LiDAR Projected Volume (LPV) were generally strong (up to r = 0.86), as were correlations between CH and LiDAR Canopy Height (LCH) (up to r = 0.94). Heritability (H2) of LPV (H2 = 0.32-0.90) was observed to be greater than, or similar to, the heritability of AGB (H2 = 0.12-0.78) for the majority of measurements. A similar level of heritability was observed for LCH (H2 = 0.41-0.98) and CH (H2 = 0.49-0.98). Further to this, measurements of LPV and LCH were shown to be highly repeatable when collected from either the same or opposite direction of travel. LiDAR scans were collected at a rate of 2,400 plots per hour, with the potential to further increase throughput to 7,400 plots per hour. This research demonstrates the capability of LiDAR sensors to collect high-quality, non-destructive, repeatable measurements of AGB and CH suitable for use within both breeding and research programs.

High-Throughput Phenotyping Analysis of Potted Soybean Plants Using Colorized Depth Images Based on A Proximal Platform

Canopy color and structure can strongly reflect plant functions. Color characteristics and plant height as well as canopy breadth are important aspects of the canopy phenotype of soybean plants. High-throughput phenotyping systems with imaging capabilities providing color and depth information can rapidly acquire data of soybean plants, making it possible to quantify and monitor soybean canopy development. The goal of this study was to develop a 3D imaging approach to quantitatively analyze soybean canopy development under natural light conditions. Thus, a Kinect sensor-based high-throughput phenotyping (HTP) platform was developed for soybean plant phenotyping. To calculate color traits accurately, the distortion phenomenon of color images was first registered in accordance with the principle of three primary colors and color constancy. Then, the registered color images were applied to depth images for the reconstruction of the colorized three-dimensional canopy structure. Furthermore, the 3D point cloud of soybean canopies was extracted from the background according to adjusted threshold, and each area of individual potted soybean plants in the depth images was segmented for the calculation of phenotypic traits. Finally, color indices, plant height and canopy breadth were assessed based on 3D point cloud of soybean canopies. The results showed that the maximum error of registration for the R, G, and B bands in the dataset was 1.26%, 1.09%, and 0.75%, respectively. Correlation analysis between the sensors and manual measurements yielded R2 values of 0.99, 0.89, and 0.89 for plant height, canopy breadth in the west-east (W–E) direction, and canopy breadth in the north-south (N–S) direction, and R2 values of 0.82, 0.79, and 0.80 for color indices h, s, and i, respectively. Given these results, the proposed approaches provide new opportunities for the identification of the quantitative traits that control canopy structure in genetic/genomic studies or for soybean yield prediction in breeding programs.

Leveraging Image Analysis for High-Throughput Plant Phenotyping

The complex interaction between a genotype and its environment controls the biophysical properties of a plant, manifested in observable traits, i.e., plant's phenome, which influences resources acquisition, performance, and yield. High-throughput automated image-based plant phenotyping refers to the sensing and quantifying plant traits non-destructively by analyzing images captured at regular intervals and with precision. While phenomic research has drawn significant attention in the last decade, extracting meaningful and reliable numerical phenotypes from plant images especially by considering its individual components, e.g., leaves, stem, fruit, and flower, remains a critical bottleneck to the translation of advances of phenotyping technology into genetic insights due to various challenges including lighting variations, plant rotations, and self-occlusions. The paper provides (1) a framework for plant phenotyping in a multimodal, multi-view, time-lapsed, high-throughput imaging system; (2) a taxonomy of phenotypes that may be derived by image analysis for better understanding of morphological structure and functional processes in plants; (3) a brief discussion on publicly available datasets to encourage algorithm development and uniform comparison with the state-of-the-art methods; (4) an overview of the state-of-the-art image-based high-throughput plant phenotyping methods; and (5) open problems for the advancement of this research field.

Comparing Nadir and Multi-Angle View Sensor Technologies for Measuring in-Field Plant Height of Upland Cotton

Plant height is a morphological characteristic of plant growth that is a useful indicator of plant stress resulting from water and nutrient deficit. While height is a relatively simple trait, it can be difficult to measure accurately, especially in crops with complex canopy architectures like cotton. This paper describes the deployment of four nadir view ultrasonic transducers (UTs), two light detection and ranging (LiDAR) systems, and an unmanned aerial system (UAS) with a digital color camera to characterize plant height in an upland cotton breeding trial. The comparison of the UTs with manual measurements demonstrated that the Honeywell and Pepperl+Fuchs sensors provided more precise estimates of plant height than the MaxSonar and db3 Pulsar sensors. Performance of the multi-angle view LiDAR and UAS technologies demonstrated that the UAS derived 3-D point clouds had stronger correlations (0.980) with the UTs than the proximal LiDAR sensors. As manual measurements require increased time and labor in large breeding trials and are prone to human error reducing repeatability, UT and UAS technologies are an efficient and effective means of characterizing cotton plant height.

Calculation Method for Phenotypic Traits Based on the 3D Reconstruction of Maize Canopies

A reasonable plant type is an essential factor for improving canopy structure, ensuring a reasonable expansion of the leaf area index and obtaining a high-quality spatial distribution of light. It is of great significance in promoting effective selection of the ecological breeding index and production practices for maize. In this study, a method for calculating the phenotypic traits of the maize canopy in three-dimensional (3D) space was proposed, focusing on the problems existing in traditional measurement methods in maize morphological structure research, such as their complex procedures and relatively large error margins. Specifically, the whole maize plant was first scanned with a FastSCAN hand-held scanner to obtain 3D point cloud data for maize. Subsequently, the raw point clouds were simplified by the grid method, and the effect of noise on the quality of the point clouds in maize canopies was further denoised by bilateral filtering. In the last step, the 3D structure of the maize canopy was reconstructed. In accordance with the 3D reconstruction of the maize canopy, the phenotypic traits of the maize canopy, such as plant height, stem diameter and canopy breadth, were calculated by means of a fitting sphere and a fitting cylinder. Thereafter, multiple regression analysis was carried out, focusing on the calculated data and the actual measured data to verify the accuracy of the calculation method proposed in this study. The corresponding results showed that the calculated values of plant height, stem diameter and plant width based on 3D scanning were highly correlated with the actual measured data, and the determinant coefficients R² were 0.9807, 0.8907 and 0.9562, respectively. In summary, the method proposed in this study can accurately measure the phenotypic traits of maize. Significantly, these research findings provide technical support for further research on the phenotypic traits of other crops and on variety breeding.

Development of a Peanut Canopy Measurement System Using a Ground-Based LiDAR Sensor

Plant architecture characteristics contribute significantly to the microclimate within peanut canopies, affecting weed suppression as well as incidence and severity of foliar and soil-borne diseases. However, plant canopy architecture is difficult to measure and describe quantitatively. In this study, a ground-based LiDAR sensor was used to scan rows of peanut plants in the field, and a data processing and analysis algorithm was developed to extract feature indices to describe the peanut canopy architecture. A data acquisition platform was constructed to carry the ground-based LiDAR and an RGB camera during field tests. An experimental field was established with three peanut cultivars at Oklahoma State University's Caddo Research Station in Fort Cobb, OK in May and the data collections were conducted once each month from July to September 2015. The ground-based LiDAR used for this research was a line-scan laser scanner with a scan-angle of 100°, an angle resolution of 0.25°, and a scanning speed of 53 ms. The collected line-scanned data were processed using the developed image processing algorithm. The canopy height, width, and shape/density were evaluated. Euler number, entropy, cluster count, and mean number of connected objects were extracted from the image and used to describe the shape of the peanut canopies. The three peanut cultivars were then classified using the shape features and indices. A high correlation was also observed between the LiDAR and ground-truth measurements for plant height. This approach should be useful for phenotyping peanut germplasm for canopy architecture.

LiDARPheno - A Low-Cost LiDAR-Based 3D Scanning System for Leaf Morphological Trait Extraction

The ever-growing world population brings the challenge for food security in the current world. The gene modification tools have opened a new era for fast-paced research on new crop identification and development. However, the bottleneck in the plant phenotyping technology restricts the alignment in geno-pheno development as phenotyping is the key for the identification of potential crop for improved yield and resistance to the changing environment. Various attempts to making the plant phenotyping a "high-throughput" have been made while utilizing the existing sensors and technology. However, the demand for 'good' phenotypic information for linkage to the genome in understanding the gene-environment interactions is still a bottleneck in the plant phenotyping technologies. Moreover, the available technologies and instruments are inaccessible, expensive, and sometimes bulky. This work attempts to address some of the critical problems, such as exploration and development of a low-cost LiDAR-based platform for phenotyping the plants in-lab and in-field. A low-cost LiDAR-based system design, LiDARPheno, is introduced in this work to assess the feasibility of the inexpensive LiDAR sensor in the leaf trait (length, width, and area) extraction. A detailed design of the LiDARPheno, based on low-cost and off-the-shelf components and modules, is presented. Moreover, the design of the firmware to control the hardware setup of the system and the user-level python-based script for data acquisition is proposed. The software part of the system utilizes the publicly available libraries and Application Programming Interfaces (APIs), making it easy to implement the system by a non-technical user. The LiDAR data analysis methods are presented, and algorithms for processing the data and extracting the leaf traits are developed. The processing includes conversion, cleaning/filtering, segmentation and trait extraction from the LiDAR data. Experiments on indoor plants and canola plants were performed for the development and validation of the methods for estimation of the leaf traits. The results of the LiDARPheno based trait extraction are compared with the SICK LMS400 (a commercial 2D LiDAR) to assess the performance of the developed system.

A Systematic Review of Perception System and Simulators for Autonomous Vehicles Research

This paper presents a systematic review of the perception systems and simulators for autonomous vehicles (AV). This work has been divided into three parts. In the first part, perception systems are categorized as environment perception systems and positioning estimation systems. The paper presents the physical fundamentals, principle functioning, and electromagnetic spectrum used to operate the most common sensors used in perception systems (ultrasonic, RADAR, LiDAR, cameras, IMU, GNSS, RTK, etc.). Furthermore, their strengths and weaknesses are shown, and the quantification of their features using spider charts will allow proper selection of different sensors depending on 11 features. In the second part, the main elements to be taken into account in the simulation of a perception system of an AV are presented. For this purpose, the paper describes simulators for model-based development, the main game engines that can be used for simulation, simulators from the robotics field, and lastly simulators used specifically for AV. Finally, the current state of regulations that are being applied in different countries around the world on issues concerning the implementation of autonomous vehicles is presented.

Wheat Height Estimation Using LiDAR in Comparison to Ultrasonic Sensor and UAS

As one of the key crop traits, plant height is traditionally evaluated manually, which can be slow, laborious and prone to error. Rapid development of remote and proximal sensing technologies in recent years allows plant height to be estimated in more objective and efficient fashions, while research regarding direct comparisons between different height measurement methods seems to be lagging. In this study, a ground-based multi-sensor phenotyping system equipped with ultrasonic sensors and light detection and ranging (LiDAR) was developed. Canopy heights of 100 wheat plots were estimated five times during a season by the ground phenotyping system and an unmanned aircraft system (UAS), and the results were compared to manual measurements. Overall, LiDAR provided the best results, with a root-mean-square error (RMSE) of 0.05 m and an R² of 0.97. UAS obtained reasonable results with an RMSE of 0.09 m and an R² of 0.91. Ultrasonic sensors did not perform well due to our static measurement style. In conclusion, we suggest LiDAR and UAS are reliable alternative methods for wheat height evaluation.

Three-Dimensional Reconstruction of Soybean Canopies Using Multisource Imaging for Phenotyping Analysis

Geometric three-dimensional (3D) reconstruction has emerged as a powerful tool for plant phenotyping and plant breeding. Although laser scanning is one of the most intensely used sensing techniques for 3D reconstruction projects, it still has many limitations, such as the high investment cost. To overcome such limitations, in the present study, a low-cost, novel, and efficient imaging system consisting of a red-green-blue (RGB) camera and a photonic mixer detector (PMD) was developed, and its usability for plant phenotyping was demonstrated via a 3D reconstruction of a soybean plant that contains color information. To reconstruct soybean canopies, a density-based spatial clustering of applications with noise (DBSCAN) algorithm was used to extract canopy information from the raw 3D point cloud. Principal component analysis (PCA) and iterative closest point (ICP) algorithms were then used to register the multisource images for the 3D reconstruction of a soybean plant from both the side and top views. We then assessed phenotypic traits such as plant height and the greenness index based on the deviations of test samples. The results showed that compared with manual measurements, the side view-based assessments yielded a determination coefficient (R2) of 0.9890 for the estimation of soybean height and a R2 of 0.6059 for the estimation of soybean canopy greenness index; the top view-based assessment yielded a R2 of 0.9936 for the estimation of soybean height and a R2 of 0.8864 for the estimation of soybean canopy greenness. Together, the results indicated that an assembled 3D imaging device applying the algorithms developed in this study could be used as a reliable and robust platform for plant phenotyping, and potentially for automated and high-throughput applications under both natural light and indoor conditions.

Translating High-Throughput Phenotyping into Genetic Gain

Inability to efficiently implement high-throughput field phenotyping is increasingly perceived as a key component that limits genetic gain in breeding programs. Field phenotyping must be integrated into a wider context than just choosing the correct selection traits, deployment tools, evaluation platforms, or basic data-management methods. Phenotyping means more than conducting such activities in a resource-efficient manner; it also requires appropriate trial management and spatial variability handling, definition of key constraining conditions prevalent in the target population of environments, and the development of more comprehensive data management, including crop modeling. This review will provide a wide perspective on how field phenotyping is best implemented. It will also outline how to bridge the gap between breeders and 'phenotypers' in an effective manner.

A Novel LiDAR-Based Instrument for High-Throughput, 3D Measurement of Morphological Traits in Maize and Sorghum

Recently, imaged-based approaches have developed rapidly for high-throughput plant phenotyping (HTPP). Imaging reduces a 3D plant into 2D images, which makes the retrieval of plant morphological traits challenging. We developed a novel LiDAR-based phenotyping instrument to generate 3D point clouds of single plants. The instrument combined a LiDAR scanner with a precision rotation stage on which an individual plant was placed. A LabVIEW program was developed to control the scanning and rotation motion, synchronize the measurements from both devices, and capture a 360° view point cloud. A data processing pipeline was developed for noise removal, voxelization, triangulation, and plant leaf surface reconstruction. Once the leaf digital surfaces were reconstructed, plant morphological traits, including individual and total leaf area, leaf inclination angle, and leaf angular distribution, were derived. The system was tested with maize and sorghum plants. The results showed that leaf area measurements by the instrument were highly correlated with the reference methods (R² > 0.91 for individual leaf area; R² > 0.95 for total leaf area of each plant). Leaf angular distributions of the two species were also derived. This instrument could fill a critical technological gap for indoor HTPP of plant morphological traits in 3D.

In-field High Throughput Phenotyping and Cotton Plant Growth Analysis Using LiDAR

Plant breeding programs and a wide range of plant science applications would greatly benefit from the development of in-field high throughput phenotyping technologies. In this study, a terrestrial LiDAR-based high throughput phenotyping system was developed. A 2D LiDAR was applied to scan plants from overhead in the field, and an RTK-GPS was used to provide spatial coordinates. Precise 3D models of scanned plants were reconstructed based on the LiDAR and RTK-GPS data. The ground plane of the 3D model was separated by RANSAC algorithm and a Euclidean clustering algorithm was applied to remove noise generated by weeds. After that, clean 3D surface models of cotton plants were obtained, from which three plot-level morphologic traits including canopy height, projected canopy area, and plant volume were derived. Canopy height ranging from 85th percentile to the maximum height were computed based on the histogram of the z coordinate for all measured points; projected canopy area was derived by projecting all points on a ground plane; and a Trapezoidal rule based algorithm was proposed to estimate plant volume. Results of validation experiments showed good agreement between LiDAR measurements and manual measurements for maximum canopy height, projected canopy area, and plant volume, with R2-values of 0.97, 0.97, and 0.98, respectively. The developed system was used to scan the whole field repeatedly over the period from 43 to 109 days after planting. Growth trends and growth rate curves for all three derived morphologic traits were established over the monitoring period for each cultivar. Overall, four different cultivars showed similar growth trends and growth rate patterns. Each cultivar continued to grow until ~88 days after planting, and from then on varied little. However, the actual values were cultivar specific. Correlation analysis between morphologic traits and final yield was conducted over the monitoring period. When considering each cultivar individually, the three traits showed the best correlations with final yield during the period between around 67 and 109 days after planting, with maximum R2-values of up to 0.84, 0.88, and 0.85, respectively. The developed system demonstrated relatively high throughput data collection and analysis.

Plant phenomics: an overview of image acquisition technologies and image data analysis algorithms

The study of phenomes or phenomics has been a central part of biology. The field of automatic phenotype acquisition technologies based on images has seen an important advance in the last years. As with other high-throughput technologies, it addresses a common set of problems, including data acquisition and analysis. In this review, we give an overview of the main systems developed to acquire images. We give an in-depth analysis of image processing with its major issues and the algorithms that are being used or emerging as useful to obtain data out of images in an automatic fashion.