Low-cost tools for virtual reconstruction of traffic accident scenarios

Investigations into traffic accidents that lead to the determination of their causes and consequences are useful to all interested parties, both in the public and private sectors. One of the phases of investigation is the capture of data enabling the complete reconstruction of the accident scene, which is usually the point at which a conflict arises between the slow process of information gathering and the need to restore normal traffic flow. To reduce to a minimum the time the traffic is halted, this paper follows a methodology to reconstruct traffic accidents and puts forward a series of procedures and tools that are applicable to both large and small scenarios. The methodology uses low-cost UAV-SfM in combination with UAS aerial image capture systems and inexpensive GNSS equipment costing less than €900. This paper describes numerous tests and assessments that were carried out on four potential work scenarios (E-1 and E-2 urban roads with several intersections; E-3, an urban crossing with medium slopes; and E-4, a complex road section with different land morphologies), assessing the impact of using simple or double strip flights and the number of GCPs, their spacing distance and different distribution patterns. From the different configurations tested, the best results were achieved in those offset-type distributions where the GCPs were placed on both sides of the working area and at each end, with a spacing between 100 and 50 m and using double strip flights. Our conclusion is that the application of this protocol would be highly efficient and economical in the reconstruction of traffic accidents, provide simplicity in implementation, speed of capture and data processing, and provide reliable results quite economically and with a high degree of accuracy with RMSE values below 5 cm.

A Color- and Geometric-Feature-Based Approach for Denoising Three-Dimensional Cultural Relic Point Clouds

In the acquisition process of 3D cultural relics, it is common to encounter noise. To facilitate the generation of high-quality 3D models, we propose an approach based on graph signal processing that combines color and geometric features to denoise the point cloud. We divide the 3D point cloud into patches based on self-similarity theory and create an appropriate underlying graph with a Markov property. The features of the vertices in the graph are represented using 3D coordinates, normal vectors, and color. We formulate the point cloud denoising problem as a maximum a posteriori (MAP) estimation problem and use a graph Laplacian regularization (GLR) prior to identifying the most probable noise-free point cloud. In the denoising process, we moderately simplify the 3D point to reduce the running time of the denoising algorithm. The experimental results demonstrate that our proposed approach outperforms five competing methods in both subjective and objective assessments. It requires fewer iterations and exhibits strong robustness, effectively removing noise from the surface of cultural relic point clouds while preserving fine-scale 3D features such as texture and ornamentation. This results in more realistic 3D representations of cultural relics.

Fixed-time synchronization of complex-valued neural networks for image protection and 3D point cloud information protection

This paper studies the fixed-time synchronization (FDTS) of complex-valued neural networks (CVNNs) based on quantized intermittent control (QIC) and applies it to image protection and 3D point cloud information protection. A new controller was designed which achieved FDTS of the CVNNs, with the estimation of the convergence time not dependent on the initial state. Our approach divides the neural network into two real-valued systems and then combines the framework of the Lyapunov method to give criteria for FDTS. Applying synchronization to image protection, the image will be encrypted with a drive system sequence and decrypted with a response system sequence. The quality of image encryption and decryption depends on the synchronization error. Meanwhile, the depth image of the object is encrypted and then the 3D point cloud is reconstructed based on the decrypted depth image. This means that the 3D point cloud information is protected. Finally, simulation examples verify the efficacy of the controller and the synchronization criterion, giving results for applications in image protection and 3D point cloud information protection.

Computer-Vision-Based Sensing Technologies for Livestock Body Dimension Measurement: A Survey

Livestock's live body dimensions are a pivotal indicator of economic output. Manual measurement is labor-intensive and time-consuming, often eliciting stress responses in the livestock. With the advancement of computer technology, the techniques for livestock live body dimension measurement have progressed rapidly, yielding significant research achievements. This paper presents a comprehensive review of the recent advancements in livestock live body dimension measurement, emphasizing the crucial role of computer-vision-based sensors. The discussion covers three main aspects: sensing data acquisition, sensing data processing, and sensing data analysis. The common techniques and measurement procedures in, and the current research status of, live body dimension measurement are introduced, along with a comparative analysis of their respective merits and drawbacks. Livestock data acquisition is the initial phase of live body dimension measurement, where sensors are employed as data collection equipment to obtain information conducive to precise measurements. Subsequently, the acquired data undergo processing, leveraging techniques such as 3D vision technology, computer graphics, image processing, and deep learning to calculate the measurements accurately. Lastly, this paper addresses the existing challenges within the domain of livestock live body dimension measurement in the livestock industry, highlighting the potential contributions of computer-vision-based sensors. Moreover, it predicts the potential development trends in the realm of high-throughput live body dimension measurement techniques for livestock.

Deep learning-based tooth segmentation methods in medical imaging: A review

Deep learning approaches for tooth segmentation employ convolutional neural networks (CNNs) or Transformers to derive tooth feature maps from extensive training datasets. Tooth segmentation serves as a critical prerequisite for clinical dental analysis and surgical procedures, enabling dentists to comprehensively assess oral conditions and subsequently diagnose pathologies. Over the past decade, deep learning has experienced significant advancements, with researchers introducing efficient models such as U-Net, Mask R-CNN, and Segmentation Transformer (SETR). Building upon these frameworks, scholars have proposed numerous enhancement and optimization modules to attain superior tooth segmentation performance. This paper discusses the deep learning methods of tooth segmentation on dental panoramic radiographs (DPRs), cone-beam computed tomography (CBCT) images, intro oral scan (IOS) models, and others. Finally, we outline performance-enhancing techniques and suggest potential avenues for ongoing research. Numerous challenges remain, including data annotation and model generalization limitations. This paper offers insights for future tooth segmentation studies, potentially facilitating broader clinical adoption.

Human Movement Recognition Based on 3D Point Cloud Spatiotemporal Information from Millimeter-Wave Radar

Human movement recognition is the use of perceptual technology to collect some of the limb or body movements presented. This practice involves the use of wireless signals, processing, and classification to identify some of the regular movements of the human body. It has a wide range of application prospects, including in intelligent pensions, remote health monitoring, and child supervision. Among the traditional human movement recognition methods, the widely used ones are video image-based recognition technology and Wi-Fi-based recognition technology. However, in some dim and imperfect weather environments, it is not easy to maintain a high performance and recognition rate for human movement recognition using video images. There is the problem of a low recognition degree for Wi-Fi recognition of human movement in the case of a complex environment. Most of the previous research on human movement recognition is based on LiDAR perception technology. LiDAR scanning using a three-dimensional static point cloud can only present the point cloud characteristics of static objects; it struggles to reflect all the characteristics of moving objects. In addition, due to its consideration of privacy and security issues, the dynamic millimeter-wave radar point cloud used in the previous study on the existing problems of human body movement recognition performance is better, with the recognition of human movement characteristics in non-line-of-sight situations as well as better protection of people's privacy. In this paper, we propose a human motion feature recognition system (PNHM) based on spatiotemporal information of the 3D point cloud of millimeter-wave radar, design a neural network based on the network PointNet++ in order to effectively recognize human motion features, and study four human motions based on the threshold method. The data set of the four movements of the human body at two angles in two experimental environments was constructed. This paper compares four standard mainstream 3D point cloud human action recognition models for the system. The experimental results show that the recognition accuracy of the human body's when walking upright can reach 94%, the recognition accuracy when moving from squatting to standing can reach 84%, that when moving from standing to sitting can reach 87%, and the recognition accuracy of falling can reach 93%.

Real-Time Road Intersection Detection in Sparse Point Cloud Based on Augmented Viewpoints Beam Model

Road intersection is a kind of important navigation landmark, while existing detection methods exhibit clear limitations in terms of their robustness and efficiency. A real-time algorithm for road intersection detection and location in large-scale sparse point clouds is proposed in this paper. Different from traditional approaches, our method establishes the augmented viewpoints beam model to perceive the road bifurcation structure. Explicitly, the spatial features from point clouds are jointly extracted in various viewpoints in front of the robot. In addition, the evaluation metrics are designed to self-assess the quality of detection results, enabling our method to optimize the detection process in real time. Considering the scarcity of datasets for intersection detection, we also collect and annotate a VLP-16 point cloud dataset specifically for intersections, called NCP-Intersection. Quantitative and qualitative experiments demonstrate that the proposed method performs favorably against the other parallel methods. Specifically, our method performs an average precision exceeding 90% and an average processing time of approximately 88 ms/frame.

Comparing Nadir and Oblique Thermal Imagery in UAV-Based 3D Crop Water Stress Index Applications for Precision Viticulture with LiDAR Validation

Unmanned Aerial Vehicle (UAV) thermal imagery is rapidly becoming an essential tool in precision agriculture. Its ability to enable widespread crop status assessment is increasingly critical, given escalating water demands and limited resources, which drive the need for optimizing water use and crop yield through well-planned irrigation and vegetation management. Despite advancements in crop assessment methodologies, including the use of vegetation indices, 2D mapping, and 3D point cloud technologies, some aspects remain less understood. For instance, mission plans often capture nadir and oblique images simultaneously, which can be time- and resource-intensive, without a clear understanding of each image type's impact. This issue is particularly critical for crops with specific growth patterns, such as woody crops, which grow vertically. This research aims to investigate the role of nadir and oblique images in the generation of CWSI (Crop Water Stress Index) maps and CWSI point clouds, that is 2D and 3D products, in woody crops for precision agriculture. To this end, products were generated using Agisoft Metashape, ArcGIS Pro, and CloudCompare to explore the effects of various flight configurations on the final outcome, seeking to identify the most efficient workflow for each remote sensing product. A linear regression analysis reveals that, for generating 2D products (orthomosaics), combining flight angles is redundant, while 3D products (point clouds) are generated equally from nadir and oblique images. Volume calculations show that combining nadir and oblique flights yields the most accurate results for CWSI point clouds compared to LiDAR in terms of geometric representation (R2 = 0.72), followed by the nadir flight (R2 = 0.68), and, finally, the oblique flight (R2 = 0.54). Thus, point clouds offer a fuller perspective of the canopy. To our knowledge, this is the first time that CWSI point clouds have been used for precision viticulture, and this knowledge can aid farm managers, technicians, or UAV pilots in optimizing the capture of UAV image datasets in line with their specific goals.

Real-Time Bucket Pose Estimation Based on Deep Neural Network and Registration Using Onboard 3D Sensor

Real-time and accurate bucket pose estimation plays a vital role in improving the intelligence level of mining excavators, as the bucket is a crucial component of the excavator. Existing methods for bucket pose estimation are realized by installing multiple non-visual sensors. However, these sensors suffer from cumulative errors caused by loose connections and short service lives caused by strong vibrations. In this paper, we propose a method for bucket pose estimation based on deep neural network and registration to solve the large registration error problem caused by occlusion. Specifically, we optimize the Point Transformer network for bucket point cloud semantic segmentation, significantly improving the segmentation accuracy. We employ point cloud preprocessing and continuous frame registration to reduce the registration distance and accelerate the Fast Iterative Closest Point algorithm, enabling real-time pose estimation. By achieving precise semantic segmentation and faster registration, we effectively address the problem of intermittent pose estimation caused by occlusion. We collected our own dataset for training and testing, and the experimental results are compared with other relevant studies, validating the accuracy and effectiveness of the proposed method.

A Method for Obtaining 3D Point Cloud Data by Combining 2D Image Segmentation and Depth Information of Pigs

This paper proposes a method for automatic pig detection and segmentation using RGB-D data for precision livestock farming. The proposed method combines the enhanced YOLOv5s model with the Res2Net bottleneck structure, resulting in improved fine-grained feature extraction and ultimately enhancing the precision of pig detection and segmentation in 2D images. Additionally, the method facilitates the acquisition of 3D point cloud data of pigs in a simpler and more efficient way by using the pig mask obtained in 2D detection and segmentation and combining it with depth information. To evaluate the effectiveness of the proposed method, two datasets were constructed. The first dataset consists of 5400 images captured in various pig pens under diverse lighting conditions, while the second dataset was obtained from the UK. The experimental results demonstrated that the improved YOLOv5s_Res2Net achieved a mAP@0.5:0.95 of 89.6% and 84.8% for both pig detection and segmentation tasks on our dataset, while achieving a mAP@0.5:0.95 of 93.4% and 89.4% on the Edinburgh pig behaviour dataset. This approach provides valuable insights for improving pig management, conducting welfare assessments, and estimating weight accurately.

Extraction of soybean plant trait parameters based on SfM-MVS algorithm combined with GRNN

Soybean is an important grain and oil crop worldwide and is rich in nutritional value. Phenotypic morphology plays an important role in the selection and breeding of excellent soybean varieties to achieve high yield. Nowadays, the mainstream manual phenotypic measurement has some problems such as strong subjectivity, high labor intensity and slow speed. To address the problems, a three-dimensional (3D) reconstruction method for soybean plants based on structure from motion (SFM) was proposed. First, the 3D point cloud of a soybean plant was reconstructed from multi-view images obtained by a smartphone based on the SFM algorithm. Second, low-pass filtering, Gaussian filtering, Ordinary Least Square (OLS) plane fitting, and Laplacian smoothing were used in fusion to automatically segment point cloud data, such as individual plants, stems, and leaves. Finally, Eleven morphological traits, such as plant height, minimum bounding box volume per plant, leaf projection area, leaf projection length and width, and leaf tilt information, were accurately and nondestructively measured by the proposed an algorithm for leaf phenotype measurement (LPM). Moreover, Support Vector Machine (SVM), Back Propagation Neural Network (BP), and Back Propagation Neural Network (GRNN) prediction models were established to predict and identify soybean plant varieties. The results indicated that, compared with the manual measurement, the root mean square error (RMSE) of plant height, leaf length, and leaf width were 0.9997, 0.2357, and 0.2666 cm, and the mean absolute percentage error (MAPE) were 2.7013%, 1.4706%, and 1.8669%, and the coefficients of determination (R2) were 0.9775, 0.9785, and 0.9487, respectively. The accuracy of predicting plant species according to the six leaf parameters was highest when using GRNN, reaching 0.9211, and the RMSE was 18.3263. Based on the phenotypic traits of plants, the differences between C3, 47-6 and W82 soybeans were analyzed genetically, and because C3 was an insect-resistant line, the trait parametes (minimum box volume per plant, number of leaves, minimum size of single leaf box, leaf projection area).The results show that the proposed method can effectively extract the 3D phenotypic structure information of soybean plants and leaves without loss which has the potential using ability in other plants with dense leaves.

A fast phenotype approach of 3D point clouds of Pinus massoniana seedlings

The phenotyping of Pinus massoniana seedlings is essential for breeding, vegetation protection, resource investigation, and so on. Few reports regarding estimating phenotypic parameters accurately in the seeding stage of Pinus massoniana plants using 3D point clouds exist. In this study, seedlings with heights of approximately 15-30 cm were taken as the research object, and an improved approach was proposed to automatically calculate five key parameters. The key procedure of our proposed method includes point cloud preprocessing, stem and leaf segmentation, and morphological trait extraction steps. In the skeletonization step, the cloud points were sliced in vertical and horizontal directions, gray value clustering was performed, the centroid of the slice was regarded as the skeleton point, and the alternative skeleton point of the main stem was determined by the DAG single source shortest path algorithm. Then, the skeleton points of the canopy in the alternative skeleton point were removed, and the skeleton point of the main stem was obtained. Last, the main stem skeleton point after linear interpolation was restored, while stem and leaf segmentation was achieved. Because of the leaf morphological characteristics of Pinus massoniana, its leaves are large and dense. Even using a high-precision industrial digital readout, it is impossible to obtain a 3D model of Pinus massoniana leaves. In this study, an improved algorithm based on density and projection is proposed to estimate the relevant parameters of Pinus massoniana leaves. Finally, five important phenotypic parameters, namely plant height, stem diameter, main stem length, regional leaf length, and total leaf number, are obtained from the skeleton and the point cloud after separation and reconstruction. The experimental results showed that there was a high correlation between the actual value from manual measurement and the predicted value from the algorithm output. The accuracies of the main stem diameter, main stem length, and leaf length were 93.5%, 95.7%, and 83.8%, respectively, which meet the requirements of real applications.

Efficient Detection and Tracking of Human Using 3D LiDAR Sensor

Light Detection and Ranging (LiDAR) technology is now becoming the main tool in many applications such as autonomous driving and human-robot collaboration. Point-cloud-based 3D object detection is becoming popular and widely accepted in the industry and everyday life due to its effectiveness for cameras in challenging environments. In this paper, we present a modular approach to detect, track and classify persons using a 3D LiDAR sensor. It combines multiple principles: a robust implementation for object segmentation, a classifier with local geometric descriptors, and a tracking solution. Moreover, we achieve a real-time solution in a low-performance machine by reducing the number of points to be processed by obtaining and predicting regions of interest via movement detection and motion prediction without any previous knowledge of the environment. Furthermore, our prototype is able to successfully detect and track persons consistently even in challenging cases due to limitations on the sensor field of view or extreme pose changes such as crouching, jumping, and stretching. Lastly, the proposed solution is tested and evaluated in multiple real 3D LiDAR sensor recordings taken in an indoor environment. The results show great potential, with particularly high confidence in positive classifications of the human body as compared to state-of-the-art approaches.

Pruning Points Detection of Sweet Pepper Plants Using 3D Point Clouds and Semantic Segmentation Neural Network

Automation in agriculture can save labor and raise productivity. Our research aims to have robots prune sweet pepper plants automatically in smart farms. In previous research, we studied detecting plant parts by a semantic segmentation neural network. Additionally, in this research, we detect the pruning points of leaves in 3D space by using 3D point clouds. Robot arms can move to these positions and cut the leaves. We proposed a method to create 3D point clouds of sweet peppers by applying semantic segmentation neural networks, the ICP algorithm, and ORB-SLAM3, a visual SLAM application with a LiDAR camera. This 3D point cloud consists of plant parts that have been recognized by the neural network. We also present a method to detect the leaf pruning points in 2D images and 3D space by using 3D point clouds. Furthermore, the PCL library was used to visualize the 3D point clouds and the pruning points. Many experiments are conducted to show the method's stability and correctness.

"Canopy fingerprints" for characterizing three-dimensional point cloud data of soybean canopies

Advances in imaging hardware allow high throughput capture of the detailed three-dimensional (3D) structure of plant canopies. The point cloud data is typically post-processed to extract coarse-scale geometric features (like volume, surface area, height, etc.) for downstream analysis. We extend feature extraction from 3D point cloud data to various additional features, which we denote as 'canopy fingerprints'. This is motivated by the successful application of the fingerprint concept for molecular fingerprints in chemistry applications and acoustic fingerprints in sound engineering applications. We developed an end-to-end pipeline to generate canopy fingerprints of a three-dimensional point cloud of soybean [Glycine max (L.) Merr.] canopies grown in hill plots captured by a terrestrial laser scanner (TLS). The pipeline includes noise removal, registration, and plot extraction, followed by the canopy fingerprint generation. The canopy fingerprints are generated by splitting the data into multiple sub-canopy scale components and extracting sub-canopy scale geometric features. The generated canopy fingerprints are interpretable and can assist in identifying patterns in a database of canopies, querying similar canopies, or identifying canopies with a certain shape. The framework can be extended to other modalities (for instance, hyperspectral point clouds) and tuned to find the most informative fingerprint representation for downstream tasks. These canopy fingerprints can aid in the utilization of canopy traits at previously unutilized scales, and therefore have applications in plant breeding and resilient crop production.

Smartphone LiDAR Data: A Case Study for Numerisation of Indoor Buildings in Railway Stations

The combination of LiDAR with other technologies for numerisation is increasingly applied in the field of building, design, and geoscience, as it often brings time and cost advantages in 3D data survey processes. In this paper, the reconstruction of 3D point cloud datasets is studied, through an experimental protocol evaluation of new LiDAR sensors on smartphones. To evaluate and analyse the 3D point cloud datasets, different experimental conditions are considered depending on the acquisition mode and the type of object or surface being scanned. The conditions allowing us to obtain the most accurate data are identified and used to propose which acquisition protocol to use. This protocol seems to be the most adapted when using these LiDAR sensors to digitise complex interior buildings such as railway stations. This paper aims to propose: (i) a methodology to suggest the adaptation of an experimental protocol based on factors (distance, luminosity, surface, time, and incidence) to assess the precision and accuracy of the smartphone LiDAR sensor in a controlled environment; (ii) a comparison, both qualitative and quantitative, of smartphone LiDAR data with other traditional 3D scanner alternatives (Faro X130, VLX, and Vz400i) while considering three representative building interior environments; and (iii) a discussion of the results obtained in a controlled and a field environment, making it possible to propose recommendations for the use of the LiDAR smartphone at the end of the numerisation of the interior space of a building.

Keyframe image processing of semantic 3D point clouds based on deep learning

With the rapid development of web technologies and the popularity of smartphones, users are uploading and sharing a large number of images every day. Therefore, it is a very important issue nowadays to enable users to discover exactly the information they need in the vast amount of data and to make it possible to integrate their large amount of image material efficiently. However, traditional content-based image retrieval techniques are based on images, and there is a "semantic gap" between this and people's understanding of images. To address this "semantic gap," a keyframe image processing method for 3D point clouds is proposed, and based on this, a U-Net-based binary data stream semantic segmentation network is established for keyframe image processing of 3D point clouds in combination with deep learning techniques.

GC-MLP: Graph Convolution MLP for Point Cloud Analysis

With the objective of addressing the problem of the fixed convolutional kernel of a standard convolution neural network and the isotropy of features making 3D point cloud data ineffective in feature learning, this paper proposes a point cloud processing method based on graph convolution multilayer perceptron, named GC-MLP. Unlike traditional local aggregation operations, the algorithm generates an adaptive kernel through the dynamic learning features of points, so that it can dynamically adapt to the structure of the object, i.e., the algorithm first adaptively assigns different weights to adjacent points according to the different relationships between the different points captured. Furthermore, local information interaction is then performed with the convolutional layers through a weight-sharing multilayer perceptron. Experimental results show that, under different task benchmark datasets (including ModelNet40 dataset, ShapeNet Part dataset, S3DIS dataset), our proposed algorithm achieves state-of-the-art for both point cloud classification and segmentation tasks.

Robust Feature Matching for 3D Point Clouds with Progressive Consistency Voting

Feature matching for 3D point clouds is a fundamental yet challenging problem in remote sensing and 3D computer vision. However, due to a number of nuisances, the initial feature correspondences generated by matching local keypoint descriptors may contain many outliers (incorrect correspondences). To remove outliers, this paper presents a robust method called progressive consistency voting (PCV). PCV aims at assigning a reliable confidence score to each correspondence such that reasonable correspondences can be achieved by simply finding top-scored ones. To compute the confidence score, we suggest fully utilizing the geometric consistency cue between correspondences and propose a voting-based scheme. In addition, we progressively mine convincing voters from the initial correspondence set and optimize the scoring result by considering top-scored correspondences at the last iteration. Experiments on several standard datasets verify that PCV outperforms five state-of-the-art methods under almost all tested conditions and is robust to noise, data decimation, clutter, occlusion, and data modality change. We also apply PCV to point cloud registration and show that it can significantly improve the registration performance.

Improved RANSAC Point Cloud Spherical Target Detection and Parameter Estimation Method Based on Principal Curvature Constraint

Spherical targets are widely used in coordinate unification of large-scale combined measurements. Through its central coordinates, scanned point cloud data from different locations can be converted into a unified coordinate reference system. However, point cloud sphere detection has the disadvantages of errors and slow detection time. For this reason, a novel method of spherical object detection and parameter estimation based on an improved random sample consensus (RANSAC) algorithm is proposed. The method is based on the RANSAC algorithm. Firstly, the principal curvature of point cloud data is calculated. Combined with the k-d nearest neighbor search algorithm, the principal curvature constraint of random sampling points is implemented to improve the quality of sample points selected by RANSAC and increase the detection speed. Secondly, the RANSAC method is combined with the total least squares method. The total least squares method is used to estimate the inner point set of spherical objects obtained by the RANSAC algorithm. The experimental results demonstrate that the method outperforms the conventional RANSAC algorithm in terms of accuracy and detection speed in estimating sphere parameters.

Surface Defect Detection of Cabbage Based on Curvature Features of 3D Point Cloud

The dents and cracks of cabbage caused by mechanical damage during transportation have a direct impact on both commercial value and storage time. In this study, a method for surface defect detection of cabbage is proposed based on the curvature feature of the 3D point cloud. First, the red-green-blue (RGB) images and depth images are collected using a RealSense-D455 depth camera for 3D point cloud reconstruction. Then, the region of interest (ROI) is extracted by statistical filtering and Euclidean clustering segmentation algorithm, and the 3D point cloud of cabbage is segmented from background noise. Then, the curvature features of the 3D point cloud are calculated using the estimated normal vector based on the least square plane fitting method. Finally, the curvature threshold is determined according to the curvature characteristic parameters, and the surface defect type and area can be detected. The flat-headed cabbage and round-headed cabbage are selected to test the surface damage of dents and cracks. The test results show that the average detection accuracy of this proposed method is 96.25%, in which, the average detection accuracy of dents is 93.3% and the average detection accuracy of cracks is 96.67%, suggesting high detection accuracy and good adaptability for various cabbages. This study provides important technical support for automatic and non-destructive detection of cabbage surface defects.

Optimization and Evaluation of Sensor Angles for Precise Assessment of Architectural Traits in Peach Trees

Fruit industries play a significant role in many aspects of global food security. They provide recognized vitamins, antioxidants, and other nutritional supplements packed in fresh fruits and other processed commodities such as juices, jams, pies, and other products. However, many fruit crops including peaches (Prunus persica (L.) Batsch) are perennial trees requiring dedicated orchard management. The architectural and morphological traits of peach trees, notably tree height, canopy area, and canopy crown volume, help to determine yield potential and precise orchard management. Thus, the use of unmanned aerial vehicles (UAVs) coupled with RGB sensors can play an important role in the high-throughput acquisition of data for evaluating architectural traits. One of the main factors that define data quality are sensor imaging angles, which are important for extracting architectural characteristics from the trees. In this study, the goal was to optimize the sensor imaging angles to extract the precise architectural trait information by evaluating the integration of nadir and oblique images. A UAV integrated with an RGB imaging sensor at three different angles (90°, 65°, and 45°) and a 3D light detection and ranging (LiDAR) system was used to acquire images of peach trees located at the Washington State University’s Tukey Horticultural Orchard, Pullman, WA, USA. A total of four approaches, comprising the use of 2D data (from UAV) and 3D point cloud (from UAV and LiDAR), were utilized to segment and measure the individual tree height and canopy crown volume. Overall, the features extracted from the images acquired at 45° and integrated nadir and oblique images showed a strong correlation with the ground reference tree height data, while the latter was highly correlated with canopy crown volume. Thus, selection of the sensor angle during UAV flight is critical for improving the accuracy of extracting architectural traits and may be useful for further precision orchard management.

A Shape Reconstruction and Measurement Method for Spherical Hedges Using Binocular Vision

The center coordinate and radius of the spherical hedges are the basic phenotypic features for automatic pruning. A binocular vision-based shape reconstruction and measurement system for front-end vision information gaining are built in this paper. Parallel binocular cameras are used as the detectors. The 2D coordinate sequence of target spherical hedges is obtained by region segmentation and object extraction process. Then, a stereo correcting algorithm is conducted to keep two cameras to be parallel. Also, an improved semi-global block matching (SGBM) algorithm is studied to get a disparity map. According to the disparity map and parallel structure of the binocular vision system, the 3D point cloud of the target is obtained. Based on this, the center coordinate and radius of the spherical hedges can be measured. Laboratory and outdoor tests on shape reconstruction and measurement are conducted. In the detection range of 2,000-2,600 mm, laboratory test shows that the average error and average relative error of standard spherical hedges radius are 1.58 mm and 0.53%, respectively; the average location deviation of the center coordinate of spherical hedges is 15.92 mm. The outdoor test shows that the average error and average relative error of spherical hedges radius by the proposed system are 4.02 mm and 0.44%, respectively; the average location deviation of the center coordinate of spherical hedges is 18.29 mm. This study provides important technical support for phenotypic feature detection in the study of automatic trimming.

Accurate Real-Time Localization Estimation in Underground Mine Environments Based on a Distance-Weight Map (DWM)

The precise localization of an underground mine environment is key to achieving unmanned and intelligent underground mining. However, in an underground environment, GPS is unavailable, there are variable and often poor lighting conditions, there is visual aliasing in long tunnels, and the occurrence of airborne dust and water, presenting great difficulty for localization. We demonstrate a high-precision, real-time, without-infrastructure underground localization method based on 3D LIDAR. The underground mine environment map was constructed based on GICP-SLAM, and inverse distance weighting (IDW) was first proposed to implement error correction based on point cloud mapping called a distance-weight map (DWM). The map was used for the localization of the underground mine environment for the first time. The approach combines point cloud frames matching and DWM matching in an unscented Kalman filter fusion process. Finally, the localization method was tested in four underground scenes, where a spatial localization error of 4 cm and 60 ms processing time per frame were obtained. We also analyze the impact of the initial pose and point cloud segmentation with respect to localization accuracy. The results showed that this new algorithm can realize low-drift, real-time localization in an underground mine environment.

Incomplete Region Estimation and Restoration of 3D Point Cloud Human Face Datasets

Owing to imperfect scans, occlusions, low reflectance of the scanned surface, and packet loss, there may be several incomplete regions in the 3D point cloud dataset. These missing regions can degrade the performance of recognition, classification, segmentation, or upsampling methods in point cloud datasets. In this study, we propose a new approach to estimate the incomplete regions of 3D point cloud human face datasets using the masking method. First, we perform some preprocessing on the input point cloud, such as rotation in the left and right angles. Then, we project the preprocessed point cloud onto a 2D surface and generate masks. Finally, we interpolate the 2D projection and the mask to produce the estimated point cloud. We also designed a deep learning model to restore the estimated point cloud to improve its quality. We use chamfer distance (CD) and hausdorff distance (HD) to evaluate the proposed method on our own human face and large-scale facial model (LSFM) datasets. The proposed method achieves an average CD and HD results of 1.30 and 21.46 for our own and 1.35 and 9.08 for the LSFM datasets, respectively. The proposed method shows better results than the existing methods.

Hole Repairing Algorithm for 3D Point Cloud Model of Symmetrical Objects Grasped by the Manipulator

For the engineering application of manipulator grasping objects, mechanical arm occlusion and limited imaging angle produce various holes in the reconstructed 3D point clouds of objects. Acquiring a complete point cloud model of the grasped object plays a very important role in the subsequent task planning of the manipulator. This paper proposes a method with which to automatically detect and repair the holes in the 3D point cloud model of symmetrical objects grasped by the manipulator. With the established virtual camera coordinate system and boundary detection, repair and classification of holes, the closed boundaries for the nested holes were detected and classified into two kinds, which correspond to the mechanical claw holes caused by mechanical arm occlusion and the missing surface produced by limited imaging angle. These two kinds of holes were repaired based on surface reconstruction and object symmetry. Experiments on simulated and real point cloud models demonstrate that our approach outperforms the other state-of-the-art 3D point cloud hole repair algorithms.

Identifying and Characterizing Conveyor Belt Longitudinal Rip by 3D Point Cloud Processing

Real-time and accurate longitudinal rip detection of a conveyor belt is crucial for the safety and efficiency of an industrial haulage system. However, the existing longitudinal detection methods possess drawbacks, often resulting in false alarms caused by tiny scratches on the belt surface. A method of identifying the longitudinal rip through three-dimensional (3D) point cloud processing is proposed to solve this issue. Specifically, the spatial point data of the belt surface are acquired by a binocular line laser stereo vision camera. Within these data, the suspected points induced by the rips and scratches were extracted. Subsequently, a clustering and discrimination mechanism was employed to distinguish the rips and scratches, and only the rip information was used as alarm criterion. Finally, the direction and maximum width of the rip can be effectively characterized in 3D space using the principal component analysis (PCA) method. This method was tested in practical experiments, and the experimental results indicate that this method can identify the longitudinal rip accurately in real time and simultaneously characterize it. Thus, applying this method can provide a more effective and appropriate solution to the identification scenes of longitudinal rip and other similar defects.

Comprehensive mPoint: A Method for 3D Point Cloud Generation of Human Bodies Utilizing FMCW MIMO mm-Wave Radar

In this paper, comprehensive mPoint, a method for generating 3D (range, azimuth, and elevation) point cloud of human targets using a Frequency-Modulated Continuous Wave (FMCW) signal and Multi-Input Multi-Output (MIMO) millimeter wave radar is proposed. Distinct from the TI-mPoint method proposed by TI technology, a comprehensive mPoint method considering both the static and dynamic characteristics of radar reflected signals is utilized to generate a high precision point cloud, resulting in more comprehensive information of the target being detected. The radar possessing 60–64 GHz FMCW signal with two sets of different dimensional antennas is utilized in order to experimentally verify the results of the methodology. By using the proposed process, the point cloud data of human targets can be obtained based on six different postures of the underlying human body. The human posture cube and point cloud accuracy rates are defined in the paper in order to quantitively and qualitatively evaluate the quality of the generated point cloud. Benefitting from the proposed comprehensive mPoint, evidence shows that the point number and the accuracy rate of the generated point cloud compared with those from the popular TI-mPoint can be largely increased by 86% and 42%, respectively. In addition, the noise level of multipath reflection can be effectively reduced. Moreover, the length of the algorithm running time is only 1.6% longer than that of the previous method as a slight tradeoff.

Progressive Deep Learning Framework for Recognizing 3D Orientations and Object Class Based on Point Cloud Representation

Deep learning approaches to estimating full 3D orientations of objects, in addition to object classes, are limited in their accuracies, due to the difficulty in learning the continuous nature of three-axis orientation variations by regression or classification with sufficient generalization. This paper presents a novel progressive deep learning framework, herein referred to as 3D POCO Net, that offers high accuracy in estimating orientations about three rotational axes yet with efficiency in network complexity. The proposed 3D POCO Net is configured, using four PointNet-based networks for independently representing the object class and three individual axes of rotations. The four independent networks are linked by in-between association subnetworks that are trained to progressively map the global features learned by individual networks one after another for fine-tuning the independent networks. In 3D POCO Net, high accuracy is achieved by combining a high precision classification based on a large number of orientation classes with a regression based on a weighted sum of classification outputs, while high efficiency is maintained by a progressive framework by which a large number of orientation classes are grouped into independent networks linked by association subnetworks. We implemented 3D POCO Net for full three-axis orientation variations and trained it with about 146 million orientation variations augmented from the ModelNet10 dataset. The testing results show that we can achieve an orientation regression error of about 2.5° with about 90% accuracy in object classification for general three-axis orientation estimation and object classification. Furthermore, we demonstrate that a pre-trained 3D POCO Net can serve as an orientation representation platform based on which orientations as well as object classes of partial point clouds from occluded objects are learned in the form of transfer learning.

Unsupervised learning of reference bony shapes for orthognathic surgical planning with a surface deformation network

PURPOSE

The purpose of this study was to reduce the experience dependence during the orthognathic surgical planning that involves virtually simulating the corrective procedure for jaw deformities.

METHODS

We introduce a geometric deep learning framework for generating reference facial bone shape models for objective guidance in surgical planning. First, we propose a surface deformation network to warp a patient's deformed bone to a set of normal bones for generating a dictionary of patient-specific normal bony shapes. Subsequently, sparse representation learning is employed to estimate a reference shape model based on the dictionary.

RESULTS

We evaluated our method on a clinical dataset containing 24 patients, and compared it with a state-of-the-art method that relies on landmark-based sparse representation. Our method yields significantly higher accuracy than the competing method for estimating normal jaws and maintains the midfaces of patients' facial bones as well as the conventional way.

CONCLUSIONS

Experimental results indicate that our method generates accurate shape models that meet clinical standards.

Three-Dimensional Linear Restoration of a Tunnel Based on Measured Track and Uncontrolled Mobile Laser Scanning

Traditional precision measurement adopts discrete artificial static observation, which cannot meet the demands of the dynamic, continuous, fine and high-precision holographic measurement of large-scale infrastructure construction and complex operation and maintenance management. Due to its advantages of fast, accurate and convenient measurement, mobile laser scanning technology is becoming a popular technology in the maintenance and measurement of infrastructure construction such as tunnels. However, in some environments without satellite signals, such as indoor areas and underground spaces, it is difficult to obtain 3D data by means of mobile measurement technology. This paper proposes a method to restore the linear of the point cloud obtained by mobile laser scanning based on the measured track center line. In this paper, the measured track position is interpolated with a cubic spline to calculate the translations, and the rotation parameters are calculated by combining the simulation design data. The point cloud of the cross-section of the tunnel under the local coordinate system is converted to the absolute coordinate system to calculate the tunnel line. In addition, the method is verified by experiments combined with the subway tunnel data, and the overall point error can be controlled to within 0.1 m. The average deviation in the horizontal direction is 0.0551 m, and that in the vertical direction is 0.0274 m. Compared with the previous methods, this method can effectively avoid the obvious deformation of the tunnel and the sharp increase in the error, and can process the tunnel point cloud data more accurately and quickly. It also provides better data support for subsequent tunnel analysis such as 3D display, completion survey, systematic hazard management and so on.

Enhanced facial expression recognition using 3D point sets and geometric deep learning

Facial expression recognition plays an essential role in human conversation and human-computer interaction. Previous research studies have recognized facial expressions mainly based on 2D image processing requiring sensitive feature engineering and conventional machine learning approaches. The purpose of the present study was to recognize facial expressions by applying a new class of deep learning called geometric deep learning directly on 3D point cloud data. Two databases (Bosphorus and SIAT-3DFE) were used. The Bosphorus database includes sixty-five subjects with seven basic expressions (i.e., anger, disgust, fearness, happiness, sadness, surprise, and neutral). The SIAT-3DFE database has 150 subjects and 4 basic facial expressions (neutral, happiness, sadness, and surprise). First, preprocessing procedures such as face center cropping, data augmentation, and point cloud denoising were applied on 3D face scans. Then, a geometric deep learning model called PointNet++ was applied. A hyperparameter tuning process was performed to find the optimal model parameters. Finally, the developed model was evaluated using the recognition rate and confusion matrix. The facial expression recognition accuracy on the Bosphorus database was 69.01% for 7 expressions and could reach 85.85% when recognizing five specific expressions (anger, disgust, happiness, surprise, and neutral). The recognition rate was 78.70% with the SIAT-3DFE database. The present study suggested that 3D point cloud could be directly processed for facial expression recognition by using geometric deep learning approach. In perspectives, the developed model will be applied for facial palsy patients to guide and optimize the functional rehabilitation program.

Label3DMaize: toolkit for 3D point cloud data annotation of maize shoots

Background

The 3D point cloud is the most direct and effective data form for studying plant structure and morphology. In point cloud studies, the point cloud segmentation of individual plants to organs directly determines the accuracy of organ-level phenotype estimation and the reliability of the 3D plant reconstruction. However, highly accurate, automatic, and robust point cloud segmentation approaches for plants are unavailable. Thus, the high-throughput segmentation of many shoots is challenging. Although deep learning can feasibly solve this issue, software tools for 3D point cloud annotation to construct the training dataset are lacking.

Results

We propose a top-to-down point cloud segmentation algorithm using optimal transportation distance for maize shoots. We apply our point cloud annotation toolkit for maize shoots, Label3DMaize, to achieve semi-automatic point cloud segmentation and annotation of maize shoots at different growth stages, through a series of operations, including stem segmentation, coarse segmentation, fine segmentation, and sample-based segmentation. The toolkit takes ∼4–10 minutes to segment a maize shoot and consumes 10–20% of the total time if only coarse segmentation is required. Fine segmentation is more detailed than coarse segmentation, especially at the organ connection regions. The accuracy of coarse segmentation can reach 97.2% that of fine segmentation.

Conclusion

Label3DMaize integrates point cloud segmentation algorithms and manual interactive operations, realizing semi-automatic point cloud segmentation of maize shoots at different growth stages. The toolkit provides a practical data annotation tool for further online segmentation research based on deep learning and is expected to promote automatic point cloud processing of various plants.

3D Scanning of the Forearm for Orthosis and HMI Applications

The rise of rehabilitation robotics has ignited a global investigation into the human machine interface (HMI) between device and user. Previous research on wearable robotics has primarily focused on robotic kinematics and controls but rarely on the actual design of the physical HMI (pHMI). This paper presents a data-driven statistical forearm surface model for designing a forearm orthosis in exoskeleton applications. The forearms of 6 subjects were 3D scanned in a custom-built jig to capture data in extreme pronation and supination poses, creating 3D point clouds of the forearm surface. Resulting data was characterized into a series of ellipses from 20 to 100% of the forearm length. Key ellipse parameters in the model include: normalized major and minor axis length, normalized center point location, tilt angle, and circularity ratio. Single-subject (SS) ellipse parameters were normalized with respect to forearm radiale-stylion (RS) length and circumference and then averaged over the 6 subjects. Averaged parameter profiles were fit with 3rd-order polynomials to create combined-subjects (CS) elliptical models of the forearm. CS models were created in the jig as-is (CS1) and after alignment to ellipse centers at 20 and 100% of the forearm length (CS2). Normalized curve fits of ellipse major and minor axes in model CS2 achieve R² values ranging from 0.898 to 0.980 indicating a high degree of correlation between cross-sectional size and position along the forearm. Most other parameters showed poor correlation with forearm position (0.005 < R² < 0.391) with the exception of tilt angle in pronation (0.877) and circularity in supination (0.657). Normalized RMSE of the CS2 ellipse-fit model ranged from 0.21 to 0.64% of forearm circumference and 0.22 to 0.46% of forearm length. The average and peak surface deviation between the scaled CS2 model and individual scans along the forearm varied from 0.56 to 2.86 mm (subject averages) and 3.86 to 7.16 (subject maximums), with the peak deviation occurring between 45 and 50% RS length. The developed equations allow reconstruction of a scalable 3D model that can be sized based on two user measures, RS length and forearm circumference, or based on generic arm measurements taken from existing anthropometric databases.

Semantic Point Cloud Segmentation Using Fast Deep Neural Network and DCRF

Accurate segmentation of entity categories is the critical step for 3D scene understanding. This paper presents a fast deep neural network model with Dense Conditional Random Field (DCRF) as a post-processing method, which can perform accurate semantic segmentation for 3D point cloud scene. On this basis, a compact but flexible framework is introduced for performing segmentation to the semantics of point clouds concurrently, contribute to more precise segmentation. Moreover, based on semantics labels, a novel DCRF model is elaborated to refine the result of segmentation. Besides, without any sacrifice to accuracy, we apply optimization to the original data of the point cloud, allowing the network to handle fewer data. In the experiment, our proposed method is conducted comprehensively through four evaluation indicators, proving the superiority of our method.

Optimization of 3D Point Clouds of Oilseed Rape Plants Based on Time-of-Flight Cameras

Three-dimensional (3D) structure is an important morphological trait of plants for describing their growth and biotic/abiotic stress responses. Various methods have been developed for obtaining 3D plant data, but the data quality and equipment costs are the main factors limiting their development. Here, we propose a method to improve the quality of 3D plant data using the time-of-flight (TOF) camera Kinect V2. A K-dimension (k-d) tree was applied to spatial topological relationships for searching points. Background noise points were then removed with a minimum oriented bounding box (MOBB) with a pass-through filter, while outliers and flying pixel points were removed based on viewpoints and surface normals. After being smoothed with the bilateral filter, the 3D plant data were registered and meshed. We adjusted the mesh patches to eliminate layered points. The results showed that the patches were closer. The average distance between the patches was 1.88 × 10-3 m, and the average angle was 17.64°, which were 54.97% and 48.33% of those values before optimization. The proposed method performed better in reducing noise and the local layered-points phenomenon, and it could help to more accurately determine 3D structure parameters from point clouds and mesh models.

Assessment of Stem Volume on Plots Using Terrestrial Laser Scanner: A Precision Forestry Application

Timber volume is an important asset, not only as an ecological component, but also as a key source of present and future revenues, which requires precise estimates. We used the Trimble TX8 survey-grade terrestrial laser scanner (TLS) to create a detailed 3D point cloud for extracting total tree height and diameter at breast height (1.3 m; DBH). We compared two different methods to accurately estimate total tree heights: the first method was based on a modified version of the local maxima algorithm for treetop detection, “H_TTD”, and for the second method we used the centers of stem cross-sections at stump height (30 cm), “H_TSP”. DBH was estimated by a computationally robust algebraic circle-fitting algorithm through hierarchical cluster analysis (HCA). This study aimed to assess the accuracy of these descriptors for evaluating total stem volume by comparing the results with the reference tree measurements. The difference between the estimated total stem volume from H_TTD and measured stems was 2.732 m³ for European oak and 2.971 m³ for Norway spruce; differences between the estimated volume from H_TSP and measured stems was 1.228 m³ and 2.006 m³ for European oak and Norway spruce, respectively. The coefficient of determination indicated a strong relationship between the measured and estimated total stem volumes from both height estimation methods with an R² = 0.89 for H_TTD and R² = 0.87 for H_TSP for European oak, and R² = 0.98 for both H_TTD and H_TSP for Norway spruce. Our study has demonstrated the feasibility of finer-resolution remote sensing data for semi-automatic stem volumetric modeling of small-scale studies with high accuracy as a potential advancement in precision forestry.

TSegNet: An efficient and accurate tooth segmentation network on 3D dental model

Automatic and accurate segmentation of dental models is a fundamental task in computer-aided dentistry. Previous methods can achieve satisfactory segmentation results on normal dental models; however, they fail to robustly handle challenging clinical cases such as dental models with missing, crowding, or misaligned teeth before orthodontic treatments. In this paper, we propose a novel end-to-end learning-based method, called TSegNet, for robust and efficient tooth segmentation on 3D scanned point cloud data of dental models. Our algorithm detects all the teeth using a distance-aware tooth centroid voting scheme in the first stage, which ensures the accurate localization of tooth objects even with irregular positions on abnormal dental models. Then, a confidence-aware cascade segmentation module in the second stage is designed to segment each individual tooth and resolve ambiguities caused by aforementioned challenging cases. We evaluated our method on a large-scale real-world dataset consisting of dental models scanned before or after orthodontic treatments. Extensive evaluations, ablation studies and comparisons demonstrate that our method can generate accurate tooth labels robustly in various challenging cases and significantly outperforms state-of-the-art approaches by 6.5% of Dice Coefficient, 3.0% of F1 score in term of accuracy, while achieving 20 times speedup of computational time.

Reactive Navigation on Natural Environments by Continuous Classification of Ground Traversability

Reactivity is a key component for autonomous vehicles navigating on natural terrains in order to safely avoid unknown obstacles. To this end, it is necessary to continuously assess traversability by processing on-board sensor data. This paper describes the case study of mobile robot Andabata that classifies traversable points from 3D laser scans acquired in motion of its vicinity to build 2D local traversability maps. Realistic robotic simulations with Gazebo were employed to appropriately adjust reactive behaviors. As a result, successful navigation tests with Andabata using the robot operating system (ROS) were performed on natural environments at low speeds.

Deep Learning Applied to Vegetation Identification and Removal Using Multidimensional Aerial Data

When performing structural inspection, the generation of three-dimensional (3D) point clouds is a common resource. Those are usually generated from photogrammetry or through laser scan techniques. However, a significant drawback for complete inspection is the presence of covering vegetation, hiding possible structural problems, and making difficult the acquisition of proper object surfaces in order to provide a reliable diagnostic. Therefore, this research’s main contribution is developing an effective vegetation removal methodology through the use of a deep learning structure that is capable of identifying and extracting covering vegetation in 3D point clouds. The proposed approach uses pre and post-processing filtering stages that take advantage of colored point clouds, if they are available, or operate independently. The results showed high classification accuracy and good effectiveness when compared with similar methods in the literature. After this step, if color is available, then a color filter is applied, enhancing the results obtained. Besides, the results are analyzed in light of real Structure From Motion (SFM) reconstruction data, which further validates the proposed method. This research also presented a colored point cloud library of bushes built for the work used by other studies in the field.

body2vec: 3D Point Cloud Reconstruction for Precise Anthropometry with Handheld Devices

Current point cloud extraction methods based on photogrammetry generate large amounts of spurious detections that hamper useful 3D mesh reconstructions or, even worse, the possibility of adequate measurements. Moreover, noise removal methods for point clouds are complex, slow and incapable to cope with semantic noise. In this work, we present body2vec, a model-based body segmentation tool that uses a specifically trained Neural Network architecture. Body2vec is capable to perform human body point cloud reconstruction from videos taken on hand-held devices (smartphones or tablets), achieving high quality anthropometric measurements. The main contribution of the proposed workflow is to perform a background removal step, thus avoiding the spurious points generation that is usual in photogrammetric reconstruction. A group of 60 persons were taped with a smartphone, and the corresponding point clouds were obtained automatically with standard photogrammetric methods. We used as a 3D silver standard the clean meshes obtained at the same time with LiDAR sensors post-processed and noise-filtered by expert anthropological biologists. Finally, we used as gold standard anthropometric measurements of the waist and hip of the same people, taken by expert anthropometrists. Applying our method to the raw videos significantly enhanced the quality of the results of the point cloud as compared with the LiDAR-based mesh, and of the anthropometric measurements as compared with the actual hip and waist perimeter measured by the anthropometrists. In both contexts, the resulting quality of body2vec is equivalent to the LiDAR reconstruction.

PLIN: A Network for Pseudo-LiDAR Point Cloud Interpolation

LiDAR sensors can provide dependable 3D spatial information at a low frequency (around 10 Hz) and have been widely applied in the field of autonomous driving and unmanned aerial vehicle (UAV). However, the camera with a higher frequency (around 20 Hz) has to be decreased so as to match with LiDAR in a multi-sensor system. In this paper, we propose a novel Pseudo-LiDAR interpolation network (PLIN) to increase the frequency of LiDAR sensor data. PLIN can generate temporally and spatially high-quality point cloud sequences to match the high frequency of cameras. To achieve this goal, we design a coarse interpolation stage guided by consecutive sparse depth maps and motion relationship. We also propose a refined interpolation stage guided by the realistic scene. Using this coarse-to-fine cascade structure, our method can progressively perceive multi-modal information and generate accurate intermediate point clouds. To the best of our knowledge, this is the first deep framework for Pseudo-LiDAR point cloud interpolation, which shows appealing applications in navigation systems equipped with LiDAR and cameras. Experimental results demonstrate that PLIN achieves promising performance on the KITTI dataset, significantly outperforming the traditional interpolation method and the state-of-the-art video interpolation technique.

Fast Method of Registration for 3D RGB Point Cloud with Improved Four Initial Point Pairs Algorithm

Three-dimensional (3D) point cloud registration is an important step in three-dimensional (3D) model reconstruction or 3D mapping. Currently, there are many methods for point cloud registration, but these methods are not able to simultaneously solve the problem of both efficiency and precision. We propose a fast method of global registration, which is based on RGB (Red, Green, Blue) value by using the four initial point pairs (FIPP) algorithm. First, the number of different RGB values of points in a dataset are counted and the colors in the target dataset having too few points are discarded by using a color filter. A candidate point set in the source dataset are then generated by comparing the similarity of colors between two datasets with color tolerance, and four point pairs are searched from the two datasets by using an improved FIPP algorithm. Finally, a rigid transformation matrix of global registration is calculated with total least square (TLS) and local registration with the iterative closest point (ICP) algorithm. The proposed method (RGB-FIPP) has been validated with two types of data, and the results show that it can effectively improve the speed of 3D point cloud registration while maintaining high accuracy. The method is suitable for points with RGB values.

Automatic Detection and Modeling of Underground Pipes Using a Portable 3D LiDAR System

Automatic and accurate mapping and modeling of underground infrastructure has become indispensable for several important tasks ranging from urban planning and construction to safety and hazard mitigation. However, this offers several technical and operational challenges. The aim of this work is to develop a portable automated mapping solution for the 3D mapping and modeling of underground pipe networks during renovation and installation work when the infrastructure is being laid down in open trenches. The system is used to scan the trench and then the 3D scans obtained from the system are registered together to form a 3D point cloud of the trench containing the pipe network using a modified global ICP (iterative closest point) method. In the 3D point cloud, pipe-like structures are segmented using fuzzy C-means clustering and then modeled using a nested MSAC (M-estimator SAmpling Consensus) algorithm. The proposed method is evaluated on real data pertaining to three different sites, containing several different types of pipes. We report an overall registration error of less than 7 % , an overall segmentation accuracy of 85 % and an overall modeling error of less than 5 % . The evaluated results not only demonstrate the efficacy but also the suitability of the proposed solution.

A Novel RGB-D SLAM Algorithm Based on Cloud Robotics

In this paper, we present a novel red-green-blue-depth simultaneous localization and mapping (RGB-D SLAM) algorithm based on cloud robotics, which combines RGB-D SLAM with the cloud robot and offloads the back-end process of the RGB-D SLAM algorithm to the cloud. This paper analyzes the front and back parts of the original RGB-D SLAM algorithm and improves the algorithm from three aspects: feature extraction, point cloud registration, and pose optimization. Experiments show the superiority of the improved algorithm. In addition, taking advantage of the cloud robotics, the RGB-D SLAM algorithm is combined with the cloud robot and the back-end part of the computationally intensive algorithm is offloaded to the cloud. Experimental validation is provided, which compares the cloud robotic-based RGB-D SLAM algorithm with the local RGB-D SLAM algorithm. The results of the experiments demonstrate the superiority of our framework. The combination of cloud robotics and RGB-D SLAM can not only improve the efficiency of SLAM but also reduce the robot’s price and size.

Semi-Automatic Calibration Method for a Bed-Monitoring System Using Infrared Image Depth Sensors

With the aging of society, the number of fall accidents has increased in hospitals and care facilities, and some accidents have happened around beds. To help prevent accidents, mats and clip sensors have been used in these facilities but they can be invasive, and their purpose may be misinterpreted. In recent years, research has been conducted using an infrared-image depth sensor as a bed-monitoring system for detecting a patient getting up, exiting the bed, and/or falling; however, some manual calibration was required initially to set up the sensor in each instance. We propose a bed-monitoring system that retains the infrared-image depth sensors but uses semi-automatic rather than manual calibration in each situation where it is applied. Our automated methods robustly calculate the bed region, surrounding floor, sensor location, and attitude, and can recognize the spatial position of the patient even when the sensor is attached but unconstrained. Also, we propose a means to reconfigure the spatial position considering occlusion by parts of the bed and also accounting for the gravity center of the patient’s body. Experimental results of multi-view calibration and motion simulation showed that our methods were effective for recognition of the spatial position of the patient.

Outdoor Scene Understanding Based on Multi-Scale PBA Image Features and Point Cloud Features

Outdoor scene understanding based on the results of point cloud classification plays an important role in mobile robots and autonomous vehicles equipped with a light detection and ranging (LiDAR) system. In this paper, a novel model named Panoramic Bearing Angle (PBA) images is proposed which is generated from 3D point clouds. In a PBA model, laser point clouds are projected onto the spherical surface to establish the correspondence relationship between the laser ranging point and the image pixels, and then we use the relative location relationship of the laser point in the 3D space to calculate the gray value of the corresponding pixel. To extract robust features from 3D laser point clouds, both image pyramid model and point cloud pyramid model are utilized to extract multiple-scale features from PBA images and original point clouds, respectively. A Random Forest classifier is used to accomplish feature screening on extracted high-dimensional features to obtain the initial classification results. Moreover, reclassification is carried out to correct the misclassification points by remapping the classification results into the PBA images and using superpixel segmentation, which makes full use of the contextual information between laser points. Within each superpixel block, the reclassification is carried out again based on the results of the initial classification results, so as to correct some misclassification points and improve the classification accuracy. Two datasets published by ETH Zurich and MINES ParisTech are used to test the classification performance, and the results show the precision and recall rate of the proposed algorithms.

Calculation Method for Phenotypic Traits Based on the 3D Reconstruction of Maize Canopies

A reasonable plant type is an essential factor for improving canopy structure, ensuring a reasonable expansion of the leaf area index and obtaining a high-quality spatial distribution of light. It is of great significance in promoting effective selection of the ecological breeding index and production practices for maize. In this study, a method for calculating the phenotypic traits of the maize canopy in three-dimensional (3D) space was proposed, focusing on the problems existing in traditional measurement methods in maize morphological structure research, such as their complex procedures and relatively large error margins. Specifically, the whole maize plant was first scanned with a FastSCAN hand-held scanner to obtain 3D point cloud data for maize. Subsequently, the raw point clouds were simplified by the grid method, and the effect of noise on the quality of the point clouds in maize canopies was further denoised by bilateral filtering. In the last step, the 3D structure of the maize canopy was reconstructed. In accordance with the 3D reconstruction of the maize canopy, the phenotypic traits of the maize canopy, such as plant height, stem diameter and canopy breadth, were calculated by means of a fitting sphere and a fitting cylinder. Thereafter, multiple regression analysis was carried out, focusing on the calculated data and the actual measured data to verify the accuracy of the calculation method proposed in this study. The corresponding results showed that the calculated values of plant height, stem diameter and plant width based on 3D scanning were highly correlated with the actual measured data, and the determinant coefficients R² were 0.9807, 0.8907 and 0.9562, respectively. In summary, the method proposed in this study can accurately measure the phenotypic traits of maize. Significantly, these research findings provide technical support for further research on the phenotypic traits of other crops and on variety breeding.

An Accurate Skeleton Extraction Approach From 3D Point Clouds of Maize Plants

Accurate and high-throughput determination of plant morphological traits is essential for phenotyping studies. Nowadays, there are many approaches to acquire high-quality three-dimensional (3D) point clouds of plants. However, it is difficult to estimate phenotyping parameters accurately of the whole growth stages of maize plants using these 3D point clouds. In this paper, an accurate skeleton extraction approach was proposed to bridge the gap between 3D point cloud and phenotyping traits estimation of maize plants. The algorithm first uses point cloud clustering and color difference denoising to reduce the noise of the input point clouds. Next, the Laplacian contraction algorithm is applied to shrink the points. Then the key points representing the skeleton of the plant are selected through adaptive sampling, and neighboring points are connected to form a plant skeleton composed of semantic organs. Finally, deviation skeleton points to the input point cloud are calibrated by building a step forward local coordinate along the tangent direction of the original points. The proposed approach successfully generates accurately extracted skeleton from 3D point cloud and helps to estimate phenotyping parameters with high precision of maize plants. Experimental verification of the skeleton extraction process, tested using three cultivars and different growth stages maize, demonstrates that the extracted matches the input point cloud well. Compared with 3D digitizing data-derived morphological parameters, the NRMSE of leaf length, leaf inclination angle, leaf top length, leaf azimuthal angle, leaf growth height, and plant height, estimated using the extracted plant skeleton, are 5.27, 8.37, 5.12, 4.42, 1.53, and 0.83%, respectively, which could meet the needs of phenotyping analysis. The time required to process a single maize plant is below 100 s. The proposed approach may play an important role in further maize research and applications, such as genotype-to-phenotype study, geometric reconstruction, functional structural maize modeling, and dynamic growth animation.

A Method for Broccoli Seedling Recognition in Natural Environment Based on Binocular Stereo Vision and Gaussian Mixture Model

Illumination in the natural environment is uncontrollable, and the field background is complex and changeable which all leads to the poor quality of broccoli seedling images. The colors of weeds and broccoli seedlings are close, especially under weedy conditions. The factors above have a large influence on the stability, velocity and accuracy of broccoli seedling recognition based on traditional 2D image processing technologies. The broccoli seedlings are higher than the soil background and weeds in height due to the growth advantage of transplanted crops. A method of broccoli seedling recognition in natural environments based on Binocular Stereo Vision and a Gaussian Mixture Model is proposed in this paper. Firstly, binocular images of broccoli seedlings were obtained by an integrated, portable and low-cost binocular camera. Then left and right images were rectified, and a disparity map of the rectified images was obtained by the Semi-Global Matching (SGM) algorithm. The original 3D dense point cloud was reconstructed using the disparity map and left camera internal parameters. To reduce the operation time, a non-uniform grid sample method was used for the sparse point cloud. After that, the Gaussian Mixture Model (GMM) cluster was exploited and the broccoli seedling points were recognized from the sparse point cloud. An outlier filtering algorithm based on k-nearest neighbors (KNN) was applied to remove the discrete points along with the recognized broccoli seedling points. Finally, an ideal point cloud of broccoli seedlings can be obtained, and the broccoli seedlings recognized. The experimental results show that the Semi-Global Matching (SGM) algorithm can meet the matching requirements of broccoli images in the natural environment, and the average operation time of SGM is 138 ms. The SGM algorithm is superior to the Sum of Absolute Differences (SAD) algorithm and Sum of Squared Differences (SSD) algorithms. The recognition results of Gaussian Mixture Model (GMM) outperforms K-means and Fuzzy c-means with the average running time of 51 ms. To process a pair of images with the resolution of 640×480, the total running time of the proposed method is 578 ms, and the correct recognition rate is 97.98% of 247 pairs of images. The average value of sensitivity is 85.91%. The average percentage of the theoretical envelope box volume to the measured envelope box volume is 95.66%. The method can provide a low-cost, real-time and high-accuracy solution for crop recognition in natural environment.