3D Imaging of Greenhouse Plants with an Inexpensive Binocular Stereo Vision System

Application of three-dimensional reconstruction technology in dentistry: a narrative review

BACKGROUND

Three-dimensional(3D) reconstruction technology is a method of transforming real goals into mathematical models consistent with computer logic expressions and has been widely used in dentistry, but the lack of review and summary leads to confusion and misinterpretation of information. The purpose of this review is to provide the first comprehensive link and scientific analysis of 3D reconstruction technology and dentistry to bridge the information bias between these two disciplines.

METHODS

The IEEE Xplore and PubMed databases were used for rigorous searches based on specific inclusion and exclusion criteria, supplemented by Google Academic as a complementary tool to retrieve all literature up to February 2023. We conducted a narrative review focusing on the empirical findings of the application of 3D reconstruction technology to dentistry.

RESULTS

We classify the technologies applied to dentistry according to their principles and summarize the different characteristics of each category, as well as the different application scenarios determined by these characteristics of each technique. In addition, we indicate their development prospects and worthy research directions in the field of dentistry, from individual techniques to the overall discipline of 3D reconstruction technology, respectively.

CONCLUSIONS

Researchers and clinicians should make different decisions on the choice of 3D reconstruction technology based on different objectives. The main trend in the future development of 3D reconstruction technology is the joint application of technology.

A Meta-Device for Intelligent Depth Perception

The optical illusion affects depth-sensing due to the limited and specific light-field information acquired by single-lens imaging. The incomplete depth information or visual deception would cause cognitive errors. To resolve this problem, an intelligent and compact depth-sensing meta-device that is miniaturized, integrated, and applicable for diverse scenes in all light levels is demonstrated. The compact and multifunction stereo vision system adopts an array with 3600 achromatic meta-lenses and a size of 1.2 × 1.2 mm2 to measure the depth over a 30 cm range with deep-learning support. The meta-lens array can act as multiple imaging lenses to collect light field information. It can also work with a light source as an active optical device to project a structured light. The meta-lens array can serve as the core functional component of a light-field imaging system under bright conditions or a structured-light projection system in the dark. The depth information in both ways can be analyzed and extracted by the convolutional neural network. This work provides a new avenue for the applications such as autonomous driving, machine vision, human-computer interaction, augmented reality, biometric identification, etc.

Automatic Branch-Leaf Segmentation and Leaf Phenotypic Parameter Estimation of Pear Trees Based on Three-Dimensional Point Clouds

The leaf phenotypic traits of plants have a significant impact on the efficiency of canopy photosynthesis. However, traditional methods such as destructive sampling will hinder the continuous monitoring of plant growth, while manual measurements in the field are both time-consuming and laborious. Nondestructive and accurate measurements of leaf phenotypic parameters can be achieved through the use of 3D canopy models and object segmentation techniques. This paper proposed an automatic branch-leaf segmentation pipeline based on lidar point cloud and conducted the automatic measurement of leaf inclination angle, length, width, and area, using pear canopy as an example. Firstly, a three-dimensional model using a lidar point cloud was established using SCENE software. Next, 305 pear tree branches were manually divided into branch points and leaf points, and 45 branch samples were selected as test data. Leaf points were further marked as 572 leaf instances on these test data. The PointNet++ model was used, with 260 point clouds as training input to carry out semantic segmentation of branches and leaves. Using the leaf point clouds in the test dataset as input, a single leaf instance was extracted by means of a mean shift clustering algorithm. Finally, based on the single leaf point cloud, the leaf inclination angle was calculated by plane fitting, while the leaf length, width, and area were calculated by midrib fitting and triangulation. The semantic segmentation model was tested on 45 branches, with a mean Precision_sem, mean Recall_sem, mean F1-score, and mean Intersection over Union (IoU) of branches and leaves of 0.93, 0.94, 0.93, and 0.88, respectively. For single leaf extraction, the Precision_ins, Recall_ins, and mean coverage (mCoV) were 0.89, 0.92, and 0.87, respectively. Using the proposed method, the estimated leaf inclination, length, width, and area of pear leaves showed a high correlation with manual measurements, with correlation coefficients of 0.94 (root mean squared error: 4.44°), 0.94 (root mean squared error: 0.43 cm), 0.91 (root mean squared error: 0.39 cm), and 0.93 (root mean squared error: 5.21 cm²), respectively. These results demonstrate that the method can automatically and accurately measure the phenotypic parameters of pear leaves. This has great significance for monitoring pear tree growth, simulating canopy photosynthesis, and optimizing orchard management.

A non-destructive approach for measuring rice panicle-level photosynthetic responses using 3D-image reconstruction

BACKGROUND

Our understanding of the physiological responses of rice inflorescence (panicle) to environmental stresses is limited by the challenge of accurately determining panicle photosynthetic parameters and their impact on grain yield. This is primarily due to the lack of a suitable gas exchange methodology for panicles and non-destructive methods to accurately determine panicle surface area.

RESULTS

To address these challenges, we have developed a custom panicle gas exchange cylinder compatible with the LiCor 6800 Infra-red Gas Analyzer. Accurate surface area measurements were determined using 3D panicle imaging to normalize the panicle-level photosynthetic measurements. We observed differential responses in both panicle and flag leaf for two temperate Japonica rice genotypes (accessions TEJ-1 and TEJ-2) exposed to heat stress during early grain filling. There was a notable divergence in the relative photosynthetic contribution of flag leaf and panicles for the heat-tolerant genotype (TEJ-2) compared to the sensitive genotype (TEJ-1).

CONCLUSION

The novelty of this method is the non-destructive and accurate determination of panicle area and photosynthetic parameters, enabling researchers to monitor temporal changes in panicle physiology during the reproductive development. The method is useful for panicle-level measurements under diverse environmental stresses and is sensitive enough to evaluate genotypic variation for panicle physiology and architecture in cereals with compact inflorescences.

A review on structural development and recognition–localization methods for end-effector of fruit–vegetable picking robots

The excellent performance of fruit and vegetable picking robots is usually contributed by the reasonable structure of end-effector and recognition–localization methods with high accuracy. As a result, efforts are focused on two aspects, and diverse structures of end-effector, target recognition methods as well as their combinations are yielded continuously. A good understanding for the working principle, advantages, limitations, and the adaptability in respective fields is helpful to design picking robots. Therefore, depending on different grasping ways, separating methods, structures, materials, and driving modes, main characteristics existing in traditional schemes will be depicted firstly. According to technical routes, advantages, potential applications, and challenges, underactuated manipulators and soft manipulators representing future development are then summarized systematically. Secondly, partial recognition and localization methods are also demonstrated. Specifically, current recognition manners adopting the single-feature, multi-feature fusion and deep learning are explained in view of their advantages, limitations, and successful instances. In the field of 3D localization, active vision based on the structured light, laser scanning, time of flight, and radar is reflected through the respective applications. Also, another 3D localization method called passive vision is also evaluated by advantages, limitations, the degree of automation, reconstruction effects, and the application scenario, such as monocular vision, binocular vision, and multiocular vision. Finally portrayed from structural development, recognition, and localization methods, it is possible to develop future end-effectors for fruit and vegetable picking robots with superior characteristics containing the less driving element, rigid–flexible–bionic coupling soft manipulators, simple control program, high efficiency, low damage, low cost, high versatility, and high recognition accuracy in all-season picking tasks.

Point cloud registration method for maize plants based on conical surface fitting-ICP

Reconstructing three-dimensional (3D) point cloud model of maize plants can provide reliable data for its growth observation and agricultural machinery research. The existing data collection systems and registration methods have low collection efficiency and poor registration accuracy. A point cloud registration method for maize plants based on conical surface fitting-iterative closest point (ICP) with automatic point cloud collection platform was proposed in this paper. Firstly, a Kinect V2 was selected to cooperate with an automatic point cloud collection platform to collect multi-angle point clouds. Then, the conical surface fitting algorithm was employed to fit the point clouds of the flowerpot wall to acquire the fitted rotation axis for coarse registration. Finally, the interval ICP registration algorithm was used for precise registration, and the Delaunay triangle meshing algorithm was chosen to triangulate the point clouds of maize plants. The maize plant at the flowering and kernel stage was selected for reconstruction experiments, the results show that: the full-angle registration takes 57.32 s, and the registration mean distance error is 1.98 mm. The measured value's relative errors between the reconstructed model and the material object of maize plant are controlled within 5%, the reconstructed model can replace maize plants for research.

Development of Stereo NIR-II Fluorescence Imaging System for 3D Tumor Vasculature in Small Animals

Near-infrared-II (NIR-II, 1000–1700 nm) fluorescence imaging boasts high spatial resolution and deep tissue penetration due to low light scattering, reduced photon absorption, and low tissue autofluorescence. NIR-II biological imaging is applied mainly in the noninvasive visualization of blood vessels and tumors in deep tissue. In the study, a stereo NIR-II fluorescence imaging system was developed for acquiring three-dimension (3D) images on tumor vasculature in real-time, on top of the development of fluorescent semiconducting polymer dots (IR-TPE Pdots) with ultra-bright NIR-II fluorescence (1000–1400 nm) and high stability to perform long-term fluorescence imaging. The NIR-II imaging system only consists of one InGaAs camera and a moving stage to simulate left-eye view and right-eye view for the construction of 3D in-depth blood vessel images. The system was validated with blood vessel phantom of tumor-bearing mice and was applied successfully in obtaining 3D blood vessel images with 0.6 mm- and 5 mm-depth resolution and 0.15 mm spatial resolution. The NIR-II stereo vision provides precise 3D information on the tumor microenvironment and blood vessel path.

Imaging Wheat Canopy Through Stereo Vision: Overcoming the Challenges of the Laboratory to Field Transition for Morphological Features Extraction

Stereo vision is a 3D imaging method that allows quick measurement of plant architecture. Historically, the method has mainly been developed in controlled conditions. This study identified several challenges to adapt the method to natural field conditions and propose solutions. The plant traits studied were leaf area, mean leaf angle, leaf angle distribution, and canopy height. The experiment took place in a winter wheat, Triticum aestivum L., field dedicated to fertilization trials at Gembloux (Belgium). Images were acquired thanks to two nadir cameras. A machine learning algorithm using RGB and HSV color spaces is proposed to perform soil-plant segmentation robust to light conditions. The matching between images of the two cameras and the leaf area computation was improved if the number of pixels in the image of a scene was binned from 2560 × 2048 to 1280 × 1024 pixels, for a distance of 1 m between the cameras and the canopy. Height descriptors such as median or 95th percentile of plant heights were useful to precisely compare the development of different canopies. Mean spike top height was measured with an accuracy of 97.1 %. The measurement of leaf area was affected by overlaps between leaves so that a calibration curve was necessary. The leaf area estimation presented a root mean square error (RMSE) of 0.37. The impact of wind on the variability of leaf area measurement was inferior to 3% except at the stem elongation stage. Mean leaf angles ranging from 53° to 62° were computed for the whole growing season. For each acquisition date during the vegetative stages, the variability of mean angle measurement was inferior to 1.5% which underpins that the method is precise.

PI-Plat: a high-resolution image-based 3D reconstruction method to estimate growth dynamics of rice inflorescence traits

BACKGROUND

Recent advances in image-based plant phenotyping have improved our capability to study vegetative stage growth dynamics. However, more complex agronomic traits such as inflorescence architecture (IA), which predominantly contributes to grain crop yield are more challenging to quantify and hence are relatively less explored. Previous efforts to estimate inflorescence-related traits using image-based phenotyping have been limited to destructive end-point measurements. Development of non-destructive inflorescence phenotyping platforms could accelerate the discovery of the phenotypic variation with respect to inflorescence dynamics and mapping of the underlying genes regulating critical yield components.

RESULTS

The major objective of this study is to evaluate post-fertilization development and growth dynamics of inflorescence at high spatial and temporal resolution in rice. For this, we developed the Panicle Imaging Platform (PI-Plat) to comprehend multi-dimensional features of IA in a non-destructive manner. We used 11 rice genotypes to capture multi-view images of primary panicle on weekly basis after the fertilization. These images were used to reconstruct a 3D point cloud of the panicle, which enabled us to extract digital traits such as voxel count and color intensity. We found that the voxel count of developing panicles is positively correlated with seed number and weight at maturity. The voxel count from developing panicles projected overall volumes that increased during the grain filling phase, wherein quantification of color intensity estimated the rate of panicle maturation. Our 3D based phenotyping solution showed superior performance compared to conventional 2D based approaches.

CONCLUSIONS

For harnessing the potential of the existing genetic resources, we need a comprehensive understanding of the genotype-to-phenotype relationship. Relatively low-cost sequencing platforms have facilitated high-throughput genotyping, while phenotyping, especially for complex traits, has posed major challenges for crop improvement. PI-Plat offers a low cost and high-resolution platform to phenotype inflorescence-related traits using 3D reconstruction-based approach. Further, the non-destructive nature of the platform facilitates analyses of the same panicle at multiple developmental time points, which can be utilized to explore the genetic variation for dynamic inflorescence traits in cereals.

Genetics of barley tiller and leaf development

In cereals, tillering and leaf development are key factors in the concept of crop ideotype, introduced in the 1960s to enhance crop yield, via manipulation of plant architecture. In the present review, we discuss advances in genetic analysis of barley shoot architecture, focusing on tillering, leaf size and angle. We also discuss novel phenotyping techniques, such as 2D and 3D imaging, that have been introduced in the era of phenomics, facilitating reliable trait measurement. We discuss the identification of genes and pathways that are involved in barley tillering and leaf development, highlighting key hormones involved in the control of plant architecture in barley and rice. Knowledge on genetic control of traits related to plant architecture provides useful resources for designing ideotypes for enhanced barley yield and performance.

PI-Plat: a high-resolution image-based 3D reconstruction method to estimate growth dynamics of rice inflorescence traits

BACKGROUND

Recent advances in image-based plant phenotyping have improved our capability to study vegetative stage growth dynamics. However, more complex agronomic traits such as inflorescence architecture (IA), which predominantly contributes to grain crop yield are more challenging to quantify and hence are relatively less explored. Previous efforts to estimate inflorescence-related traits using image-based phenotyping have been limited to destructive end-point measurements. Development of non-destructive inflorescence phenotyping platforms could accelerate the discovery of the phenotypic variation with respect to inflorescence dynamics and mapping of the underlying genes regulating critical yield components.

RESULTS

The major objective of this study is to evaluate post-fertilization development and growth dynamics of inflorescence at high spatial and temporal resolution in rice. For this, we developed the Panicle Imaging Platform (PI-Plat) to comprehend multi-dimensional features of IA in a non-destructive manner. We used 11 rice genotypes to capture multi-view images of primary panicle on weekly basis after the fertilization. These images were used to reconstruct a 3D point cloud of the panicle, which enabled us to extract digital traits such as voxel count and color intensity. We found that the voxel count of developing panicles is positively correlated with seed number and weight at maturity. The voxel count from developing panicles projected overall volumes that increased during the grain filling phase, wherein quantification of color intensity estimated the rate of panicle maturation. Our 3D based phenotyping solution showed superior performance compared to conventional 2D based approaches.

CONCLUSIONS

For harnessing the potential of the existing genetic resources, we need a comprehensive understanding of the genotype-to-phenotype relationship. Relatively low-cost sequencing platforms have facilitated high-throughput genotyping, while phenotyping, especially for complex traits, has posed major challenges for crop improvement. PI-Plat offers a low cost and high-resolution platform to phenotype inflorescence-related traits using 3D reconstruction-based approach. Further, the non-destructive nature of the platform facilitates analyses of the same panicle at multiple developmental time points, which can be utilized to explore the genetic variation for dynamic inflorescence traits in cereals.

Leaf Segmentation on Dense Plant Point Clouds with Facet Region Growing

Leaves account for the largest proportion of all organ areas for most kinds of plants, and are comprise the main part of the photosynthetically active material in a plant. Observation of individual leaves can help to recognize their growth status and measure complex phenotypic traits. Current image-based leaf segmentation methods have problems due to highly restricted species and vulnerability toward canopy occlusion. In this work, we propose an individual leaf segmentation approach for dense plant point clouds using facet over-segmentation and facet region growing. The approach can be divided into three steps: (1) point cloud pre-processing, (2) facet over-segmentation, and (3) facet region growing for individual leaf segmentation. The experimental results show that the proposed method is effective and efficient in segmenting individual leaves from 3D point clouds of greenhouse ornamentals such as Epipremnum aureum, Monstera deliciosa, and Calathea makoyana, and the average precision and recall are both above 90%. The results also reveal the wide applicability of the proposed methodology for point clouds scanned from different kinds of 3D imaging systems, such as stereo vision and Kinect v2. Moreover, our method is potentially applicable in a broad range of applications that aim at segmenting regular surfaces and objects from a point cloud.

Evaluating Geometric Measurement Accuracy Based on 3D Reconstruction of Automated Imagery in a Greenhouse

Geometric dimensions of plants are significant parameters for showing plant dynamic responses to environmental variations. An image-based high-throughput phenotyping platform was developed to automatically measure geometric dimensions of plants in a greenhouse. The goal of this paper was to evaluate the accuracy in geometric measurement using the Structure from Motion (SfM) method from images acquired using the automated image-based platform. Images of nine artificial objects of different shapes were taken under 17 combinations of three different overlaps in x and y directions, respectively, and two different spatial resolutions (SRs) with three replicates. Dimensions in x, y and z of these objects were measured from 3D models reconstructed using the SfM method to evaluate the geometric accuracy. A metric power of unit (POU) was proposed to combine the effects of image overlap and SR. Results showed that measurement error of dimension in z is the least affected by overlap and SR among the three dimensions and measurement error of dimensions in x and y increased following a power function with the decrease of POU (R² = 0.78 and 0.88 for x and y respectively). POUs from 150 to 300 are a preferred range to obtain reasonable accuracy and efficiency for the developed image-based high-throughput phenotyping system. As a study case, the developed system was used to measure the height of 44 plants using an optimal POU in greenhouse environment. The results showed a good agreement (R² = 92% and Root Mean Square Error = 9.4 mm) between the manual and automated method.

A High Throughput Integrated Hyperspectral Imaging and 3D Measurement System

Hyperspectral and three-dimensional measurements can obtain the intrinsic physicochemical properties and external geometrical characteristics of objects, respectively. The combination of these two kinds of data can provide new insights into objects, which has gained attention in the fields of agricultural management, plant phenotyping, cultural heritage conservation, and food production. Currently, a variety of sensors are integrated into a system to collect spectral and morphological information in agriculture. However, previous experiments were usually performed with several commercial devices on a single platform. Inadequate registration and synchronization among instruments often resulted in mismatch between spectral and 3D information of the same target. In addition, using slit-based spectrometers and point-based 3D sensors extends the working hours in farms due to the narrow field of view (FOV). Therefore, we propose a high throughput prototype that combines stereo vision and grating dispersion to simultaneously acquire hyperspectral and 3D information. Furthermore, fiber-reformatting imaging spectrometry (FRIS) is adopted to acquire the hyperspectral images. Test experiments are conducted for the verification of the system accuracy, and vegetation measurements are carried out to demonstrate its feasibility. The proposed system is an improvement in multiple data acquisition and has the potential to improve plant phenotyping.