Deep Segmentation of Point Clouds of Wheat

The improved stratified transformer for organ segmentation of Arabidopsis

Segmenting plant organs is a crucial step in extracting plant phenotypes. Despite the advancements in point-based neural networks, the field of plant point cloud segmentation suffers from a lack of adequate datasets. In this study, we addressed this issue by generating Arabidopsis models using L-system and proposing the surface-weighted sampling method. This approach enables automated point sampling and annotation, resulting in fully annotated point clouds. To create the Arabidopsis dataset, we employed Voxel Centroid Sampling and Random Sampling as point cloud downsampling methods, effectively reducing the number of points. To enhance the efficiency of semantic segmentation in plant point clouds, we introduced the Plant Stratified Transformer. This network is an improved version of the Stratified Transformer, incorporating the Fast Downsample Layer. Our improved network underwent training and testing on our dataset, and we compared its performance with PointNet++, PAConv, and the original Stratified Transformer network. For semantic segmentation, our improved network achieved mean Precision, Recall, F1-score and IoU of 84.20, 83.03, 83.61 and 73.11%, respectively. It outperformed PointNet++ and PAConv and performed similarly to the original network. Regarding efficiency, the training time and inference time were 714.3 and 597.9 ms, respectively, which were reduced by 320.9 and 271.8 ms, respectively, compared to the original network. The improved network significantly accelerated the speed of feeding point clouds into the network while maintaining segmentation performance. We demonstrated the potential of virtual plants and deep learning methods in rapidly extracting plant phenotypes, contributing to the advancement of plant phenotype research.

Using high-throughput phenotype platform MVS-Pheno to reconstruct the 3D morphological structure of wheat

It is of great significance to study the plant morphological structure for improving crop yield and achieving efficient use of resources. Three dimensional (3D) information can more accurately describe the morphological and structural characteristics of crop plants. Automatic acquisition of 3D information is one of the key steps in plant morphological structure research. Taking wheat as the research object, we propose a point cloud data-driven 3D reconstruction method that achieves 3D structure reconstruction and plant morphology parameterization at the phytomer scale. Specifically, we use the MVS-Pheno platform to reconstruct the point cloud of wheat plants and segment organs through the deep learning algorithm. On this basis, we automatically reconstructed the 3D structure of leaves and tillers and extracted the morphological parameters of wheat. The results show that the semantic segmentation accuracy of organs is 95.2%, and the instance segmentation accuracy AP50 is 0.665. The R2 values for extracted leaf length, leaf width, leaf attachment height, stem leaf angle, tiller length, and spike length were 0.97, 0.80, 1.00, 0.95, 0.99, and 0.95, respectively. This method can significantly improve the accuracy and efficiency of 3D morphological analysis of wheat plants, providing strong technical support for research in fields such as agricultural production optimization and genetic breeding.

A comparative study on point cloud down-sampling strategies for deep learning-based crop organ segmentation

The 3D crop data obtained during cultivation is of great significance to screening excellent varieties in modern breeding and improvement on crop yield. With the rapid development of deep learning, researchers have been making innovations in aspects of both data preparation and deep network design for segmenting plant organs from 3D data. Training of the deep learning network requires the input point cloud to have a fixed scale, which means all point clouds in the batch should have similar scale and contain the same number of points. A good down-sampling strategy can reduce the impact of noise and meanwhile preserve the most important 3D spatial structures. As far as we know, this work is the first comprehensive study of the relationship between multiple down-sampling strategies and the performances of popular networks for plant point clouds. Five down-sampling strategies (including FPS, RS, UVS, VFPS, and 3DEPS) are cross evaluated on five different segmentation networks (including PointNet +  + , DGCNN, PlantNet, ASIS, and PSegNet). The overall experimental results show that currently there is no strict golden rule on fixing down-sampling strategy for a specific mainstream crop deep learning network, and the optimal down-sampling strategy may vary on different networks. However, some general experience for choosing an appropriate sampling method for a specific network can still be summarized from the qualitative and quantitative experiments. First, 3DEPS and UVS are easy to generate better results on semantic segmentation networks. Second, the voxel-based down-sampling strategies may be more suitable for complex dual-function networks. Third, at 4096-point resolution, 3DEPS usually has only a small margin compared with the best down-sampling strategy at most cases, which means 3DEPS may be the most stable strategy across all compared. This study not only helps to further improve the accuracy of point cloud deep learning networks for crop organ segmentation, but also gives clue to the alignment of down-sampling strategies and a specific network.

Three-dimensional branch segmentation and phenotype extraction of maize tassel based on deep learning

BACKGROUND

The morphological structure phenotype of maize tassel plays an important role in plant growth, reproduction, and yield formation. It is an important step in the distinctness, uniformity, and stability (DUS) testing to obtain maize tassel phenotype traits. Plant organ segmentation can be achieved with high-precision and automated acquisition of maize tassel phenotype traits because of the advances in the point cloud deep learning method. However, this method requires a large number of data sets and is not robust to automatic segmentation of highly adherent organ components; thus, it should be combined with point cloud processing technology.

RESULTS

An innovative method of incomplete annotation of point cloud data was proposed for easy development of the dataset of maize tassels,and an automatic maize tassel phenotype analysis system: MaizeTasselSeg was developed. The tip feature of point cloud is trained and learned based on PointNet +  + network, and the tip point cloud of tassel branch was automatically segmented. Complete branch segmentation was realized based on the shortest path algorithm. The Intersection over Union (IoU), precision, and recall of the segmentation results were 96.29, 96.36, and 93.01, respectively. Six phenotypic traits related to morphological structure (branch count, branch length, branch angle, branch curvature, tassel volume, and dispersion) were automatically extracted from the segmentation point cloud. The squared correlation coefficients (R²) for branch length, branch angle, and branch count were 0.9897, 0.9317, and 0.9587, respectively. The root mean squared error (RMSE) for branch length, branch angle, and branch count were 0.529 cm, 4.516, and 0.875, respectively.

CONCLUSION

The proposed method provides an efficient scheme for high-throughput organ segmentation of maize tassels and can be used for the automatic extraction of phenotypic traits of maize tassel. In addition, the incomplete annotation approach provides a new idea for morphology-based plant segmentation.

Point clouds segmentation of rapeseed siliques based on sparse-dense point clouds mapping

In this study, we propose a high-throughput and low-cost automatic detection method based on deep learning to replace the inefficient manual counting of rapeseed siliques. First, a video is captured with a smartphone around the rapeseed plants in the silique stage. Feature point detection and matching based on SIFT operators are applied to the extracted video frames, and sparse point clouds are recovered using epipolar geometry and triangulation principles. The depth map is obtained by calculating the disparity of the matched images, and the dense point cloud is fused. The plant model of the whole rapeseed plant in the silique stage is reconstructed based on the structure-from-motion (SfM) algorithm, and the background is removed by using the passthrough filter. The downsampled 3D point cloud data is processed by the DGCNN network, and the point cloud is divided into two categories: sparse rapeseed canopy siliques and rapeseed stems. The sparse canopy siliques are then segmented from the original whole rapeseed siliques point cloud using the sparse-dense point cloud mapping method, which can effectively save running time and improve efficiency. Finally, Euclidean clustering segmentation is performed on the rapeseed canopy siliques, and the RANSAC algorithm is used to perform line segmentation on the connected siliques after clustering, obtaining the three-dimensional spatial position of each silique and counting the number of siliques. The proposed method was applied to identify 1457 siliques from 12 rapeseed plants, and the experimental results showed a recognition accuracy greater than 97.80%. The proposed method achieved good results in rapeseed silique recognition and provided a useful example for the application of deep learning networks in dense 3D point cloud segmentation.

Digital whole-community phenotyping: tracking morphological and physiological responses of plant communities to environmental changes in the field

Plant traits are informative for ecosystem functions and processes and help to derive general rules and predictions about responses to environmental gradients, global change and perturbations. Ecological field studies often use 'low-throughput' methods to assess plant phenotypes and integrate species-specific traits to community-wide indices. In contrast, agricultural greenhouse or lab-based studies often employ 'high-throughput phenotyping' to assess plant individuals tracking their growth or fertilizer and water demand. In ecological field studies, remote sensing makes use of freely movable devices like satellites or unmanned aerial vehicles (UAVs) which provide large-scale spatial and temporal data. Adopting such methods for community ecology on a smaller scale may provide novel insights on the phenotypic properties of plant communities and fill the gap between traditional field measurements and airborne remote sensing. However, the trade-off between spatial resolution, temporal resolution and scope of the respective study requires highly specific setups so that the measurements fit the scientific question. We introduce small-scale, high-resolution digital automated phenotyping as a novel source of quantitative trait data in ecological field studies that provides complementary multi-faceted data of plant communities. We customized an automated plant phenotyping system for its mobile application in the field for 'digital whole-community phenotyping' (DWCP), capturing the 3-dimensional structure and multispectral information of plant communities. We demonstrated the potential of DWCP by recording plant community responses to experimental land-use treatments over two years. DWCP captured changes in morphological and physiological community properties in response to mowing and fertilizer treatments and thus reliably informed about changes in land-use. In contrast, manually measured community-weighted mean traits and species composition remained largely unaffected and were not informative about these treatments. DWCP proved to be an efficient method for characterizing plant communities, complements other methods in trait-based ecology, provides indicators of ecosystem states, and may help to forecast tipping points in plant communities often associated with irreversible changes in ecosystems.

Generating 3D Multispectral Point Clouds of Plants with Fusion of Snapshot Spectral and RGB-D Images

Accurate and high-throughput plant phenotyping is important for accelerating crop breeding. Spectral imaging that can acquire both spectral and spatial information of plants related to structural, biochemical, and physiological traits becomes one of the popular phenotyping techniques. However, close-range spectral imaging of plants could be highly affected by the complex plant structure and illumination conditions, which becomes one of the main challenges for close-range plant phenotyping. In this study, we proposed a new method for generating high-quality plant 3-dimensional multispectral point clouds. Speeded-Up Robust Features and Demons was used for fusing depth and snapshot spectral images acquired at close range. A reflectance correction method for plant spectral images based on hemisphere references combined with artificial neural network was developed for eliminating the illumination effects. The proposed Speeded-Up Robust Features and Demons achieved an average structural similarity index measure of 0.931, outperforming the classic approaches with an average structural similarity index measure of 0.889 in RGB and snapshot spectral image registration. The distribution of digital number values of the references at different positions and orientations was simulated using artificial neural network with the determination coefficient (R ²) of 0.962 and root mean squared error of 0.036. Compared with the ground truth measured by ASD spectrometer, the average root mean squared error of the reflectance spectra before and after reflectance correction at different leaf positions decreased by 78.0%. For the same leaf position, the average Euclidean distances between the multiview reflectance spectra decreased by 60.7%. Our results indicate that the proposed method achieves a good performance in generating plant 3-dimensional multispectral point clouds, which is promising for close-range plant phenotyping.

Cotton plant part 3D segmentation and architectural trait extraction using point voxel convolutional neural networks

BACKGROUND

Plant architecture can influence crop yield and quality. Manual extraction of architectural traits is, however, time-consuming, tedious, and error prone. The trait estimation from 3D data addresses occlusion issues with the availability of depth information while deep learning approaches enable learning features without manual design. The goal of this study was to develop a data processing workflow by leveraging 3D deep learning models and a novel 3D data annotation tool to segment cotton plant parts and derive important architectural traits.

RESULTS

The Point Voxel Convolutional Neural Network (PVCNN) combining both point- and voxel-based representations of 3D data shows less time consumption and better segmentation performance than point-based networks. Results indicate that the best mIoU (89.12%) and accuracy (96.19%) with average inference time of 0.88 s were achieved through PVCNN, compared to Pointnet and Pointnet++. On the seven derived architectural traits from segmented parts, an R² value of more than 0.8 and mean absolute percentage error of less than 10% were attained.

CONCLUSION

This plant part segmentation method based on 3D deep learning enables effective and efficient architectural trait measurement from point clouds, which could be useful to advance plant breeding programs and characterization of in-season developmental traits. The plant part segmentation code is available at https://github.com/UGA-BSAIL/plant_3d_deep_learning .

Phenotyping of Silique Morphology in Oilseed Rape Using Skeletonization with Hierarchical Segmentation

Silique morphology is an important trait that determines the yield output of oilseed rape (Brassica napus L.). Segmenting siliques and quantifying traits are challenging because of the complicated structure of an oilseed rape plant at the reproductive stage. This study aims to develop an accurate method in which a skeletonization algorithm was combined with the hierarchical segmentation (SHS) algorithm to separate siliques from the whole plant using 3-dimensional (3D) point clouds. We combined the L1-median skeleton with the random sample consensus for iteratively extracting skeleton points and optimized the skeleton based on information such as distance, angle, and direction from neighborhood points. Density-based spatial clustering of applications with noise and weighted unidirectional graph were used to achieve hierarchical segmentation of siliques. Using the SHS, we quantified the silique number (SN), silique length (SL), and silique volume (SV) automatically based on the geometric rules. The proposed method was tested with the oilseed rape plants at the mature stage grown in a greenhouse and field. We found that our method showed good performance in silique segmentation and phenotypic extraction with R ² values of 0.922 and 0.934 for SN and total SL, respectively. Additionally, SN, total SL, and total SV had the statistical significance of correlations with the yield of a plant, with R values of 0.935, 0.916, and 0.897, respectively. Overall, the SHS algorithm is accurate, efficient, and robust for the segmentation of siliques and extraction of silique morphological parameters, which is promising for high-throughput silique phenotyping in oilseed rape breeding.

Dynamic detection of three-dimensional crop phenotypes based on a consumer-grade RGB-D camera

INTRODUCTION

Nondestructive detection of crop phenotypic traits in the field is very important for crop breeding. Ground-based mobile platforms equipped with sensors can efficiently and accurately obtain crop phenotypic traits. In this study, we propose a dynamic 3D data acquisition method in the field suitable for various crops by using a consumer-grade RGB-D camera installed on a ground-based movable platform, which can collect RGB images as well as depth images of crop canopy sequences dynamically.

METHODS

A scale-invariant feature transform (SIFT) operator was used to detect adjacent date frames acquired by the RGB-D camera to calculate the point cloud alignment coarse matching matrix and the displacement distance of adjacent images. The data frames used for point cloud matching were selected according to the calculated displacement distance. Then, the colored ICP (iterative closest point) algorithm was used to determine the fine matching matrix and generate point clouds of the crop row. The clustering method was applied to segment the point cloud of each plant from the crop row point cloud, and 3D phenotypic traits, including plant height, leaf area and projected area of individual plants, were measured.

RESULTS AND DISCUSSION

We compared the effects of LIDAR and image-based 3D reconstruction methods, and experiments were carried out on corn, tobacco, cottons and Bletilla striata in the seedling stage. The results show that the measurements of the plant height (R²= 0.9~0.96, RSME = 0.015~0.023 m), leaf area (R²= 0.8~0.86, RSME = 0.0011~0.0041 m ² ) and projected area (R² = 0.96~0.99) have strong correlations with the manual measurement results. Additionally, 3D reconstruction results with different moving speeds and times throughout the day and in different scenes were also verified. The results show that the method can be applied to dynamic detection with a moving speed up to 0.6 m/s and can achieve acceptable detection results in the daytime, as well as at night. Thus, the proposed method can improve the efficiency of individual crop 3D point cloud data extraction with acceptable accuracy, which is a feasible solution for crop seedling 3D phenotyping outdoors.

Rapid Detection of Wheat Ears in Orthophotos From Unmanned Aerial Vehicles in Fields Based on YOLOX

Wheat ears in unmanned aerial vehicles (UAV) orthophotos are characterized by occlusion, small targets, dense distribution, and complex backgrounds. Rapid identification of wheat ears in UAV orthophotos in a field environment is critical for wheat yield prediction. Three improvements were achieved based on YOLOX-m: mosaic optimized, using BiFPN structure, and attention mechanism, then ablation experiments were performed to verify the effect of each improvement. Three scene datasets were established: images were acquired during three different growing periods, at three planting densities, and under three scenarios of UAV flight heights. In ablation experiments, three improvements had increased recognition accuracies on the experimental dataset. Compared the accuracy of the standard model with our improved model on three scene datasets. Our improved model during three different periods, at three planting densities, and under three scenarios of the UAV flight height, obtaining 88.03%, 87.59%, and 87.93% accuracies, which were, respectively, 2.54%, 1.89%, and 2.15% better than the original model. The results of this study showed that the improved YOLOX-m model can achieve UAV orthophoto wheat recognition under different practical scenarios in large fields, and that the best combination were obtained images from the wheat milk stage, low planting density, and low flight altitude.