1
|
Abeyrathna RMRD, Nakaguchi VM, Liu Z, Sampurno RM, Ahamed T. 3D Camera and Single-Point Laser Sensor Integration for Apple Localization in Spindle-Type Orchard Systems. SENSORS (BASEL, SWITZERLAND) 2024; 24:3753. [PMID: 38931535 PMCID: PMC11207995 DOI: 10.3390/s24123753] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/09/2024] [Revised: 05/28/2024] [Accepted: 06/06/2024] [Indexed: 06/28/2024]
Abstract
Accurate localization of apples is the key factor that determines a successful harvesting cycle in the automation of apple harvesting for unmanned operations. In this regard, accurate depth sensing or positional information of apples is required for harvesting apples based on robotic systems, which is challenging in outdoor environments because of uneven light variations when using 3D cameras for the localization of apples. Therefore, this research attempted to overcome the effect of light variations for the 3D cameras during outdoor apple harvesting operations. Thus, integrated single-point laser sensors for the localization of apples using a state-of-the-art model, the EfficientDet object detection algorithm with an mAP@0.5 of 0.775 were used in this study. In the experiments, a RealSense D455f RGB-D camera was integrated with a single-point laser ranging sensor utilized to obtain precise apple localization coordinates for implementation in a harvesting robot. The single-point laser range sensor was attached to two servo motors capable of moving the center position of the detected apples based on the detection ID generated by the DeepSORT (online real-time tracking) algorithm. The experiments were conducted under indoor and outdoor conditions in a spindle-type apple orchard artificial architecture by mounting the combined sensor system behind a four-wheel tractor. The localization coordinates were compared between the RGB-D camera depth values and the combined sensor system under different light conditions. The results show that the root-mean-square error (RMSE) values of the RGB-D camera depth and integrated sensor mechanism varied from 3.91 to 8.36 cm and from 1.62 to 2.13 cm under 476~600 lx to 1023~1100 × 100 lx light conditions, respectively. The integrated sensor system can be used for an apple harvesting robotic manipulator with a positional accuracy of ±2 cm, except for some apples that were occluded due to leaves and branches. Further research will be carried out using changes in the position of the integrated system for recognition of the affected apples for harvesting operations.
Collapse
Affiliation(s)
- R. M. Rasika D. Abeyrathna
- Graduate School of Science and Technology, University of Tsukuba, 1-1-1 Tennodai, Tsukuba 305-8577, Japan; (R.M.R.D.A.); (V.M.N.); (Z.L.); (R.M.S.)
- Department of Agricultural Engineering, University of Peradeniya, Kandy 20400, Sri Lanka
| | - Victor Massaki Nakaguchi
- Graduate School of Science and Technology, University of Tsukuba, 1-1-1 Tennodai, Tsukuba 305-8577, Japan; (R.M.R.D.A.); (V.M.N.); (Z.L.); (R.M.S.)
| | - Zifu Liu
- Graduate School of Science and Technology, University of Tsukuba, 1-1-1 Tennodai, Tsukuba 305-8577, Japan; (R.M.R.D.A.); (V.M.N.); (Z.L.); (R.M.S.)
| | - Rizky Mulya Sampurno
- Graduate School of Science and Technology, University of Tsukuba, 1-1-1 Tennodai, Tsukuba 305-8577, Japan; (R.M.R.D.A.); (V.M.N.); (Z.L.); (R.M.S.)
- Department of Agricultural and Biosystem Engineering, Universitas Padjadjaran, Sumedang 45363, Indonesia
| | - Tofael Ahamed
- Institute of Life and Environmental Science, University of Tsukuba, 1-1-1 Tennodai, Tsukuba 305-8577, Japan
| |
Collapse
|
2
|
Pereira T, Gameiro T, Pedro J, Viegas C, Ferreira NMF. Vision System for a Forestry Navigation Machine. SENSORS (BASEL, SWITZERLAND) 2024; 24:1475. [PMID: 38475010 DOI: 10.3390/s24051475] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/18/2024] [Revised: 02/19/2024] [Accepted: 02/22/2024] [Indexed: 03/14/2024]
Abstract
This article presents the development of a vision system designed to enhance the autonomous navigation capabilities of robots in complex forest environments. Leveraging RGBD and thermic cameras, specifically the Intel RealSense 435i and FLIR ADK, the system integrates diverse visual sensors with advanced image processing algorithms. This integration enables robots to make real-time decisions, recognize obstacles, and dynamically adjust their trajectories during operation. The article focuses on the architectural aspects of the system, emphasizing the role of sensors and the formulation of algorithms crucial for ensuring safety during robot navigation in challenging forest terrains. Additionally, the article discusses the training of two datasets specifically tailored to forest environments, aiming to evaluate their impact on autonomous navigation. Tests conducted in real forest conditions affirm the effectiveness of the developed vision system. The results underscore the system's pivotal contribution to the autonomous navigation of robots in forest environments.
Collapse
Affiliation(s)
- Tiago Pereira
- Polytechnic Institute of Coimbra, Coimbra Institute of Engineering, Rua Pedro Nunes-Quinta da Nora, 3030-199 Coimbra, Portugal
| | - Tiago Gameiro
- Polytechnic Institute of Coimbra, Coimbra Institute of Engineering, Rua Pedro Nunes-Quinta da Nora, 3030-199 Coimbra, Portugal
| | - José Pedro
- ADAI (Associação para o Desenvolvimento da Aerodinâmica Industrial), Department of Mechanical Engineering, University of Coimbra, Rua Luís Reis Santos, Pólo II, 3030-788 Coimbra, Portugal
| | - Carlos Viegas
- ADAI (Associação para o Desenvolvimento da Aerodinâmica Industrial), Department of Mechanical Engineering, University of Coimbra, Rua Luís Reis Santos, Pólo II, 3030-788 Coimbra, Portugal
| | - N M Fonseca Ferreira
- Polytechnic Institute of Coimbra, Coimbra Institute of Engineering, Rua Pedro Nunes-Quinta da Nora, 3030-199 Coimbra, Portugal
- GECAD-Knowledge Research Group on Intelligent Engineering and Computing for Advanced Innovation and Development, Engineering Institute of Porto (ISEP), Polytechnic Institute of Porto (IPP), 4200-465 Porto, Portugal
| |
Collapse
|
3
|
Konecny J, Beremlijski P, Bailova M, Machacek Z, Koziorek J, Prauzek M. Industrial camera model positioned on an effector for automated tool center point calibration. Sci Rep 2024; 14:323. [PMID: 38172245 PMCID: PMC10764955 DOI: 10.1038/s41598-023-51011-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/12/2023] [Accepted: 12/29/2023] [Indexed: 01/05/2024] Open
Abstract
The study presents a novel, full model of an industrial camera suitable for robotic manipulator tool center point (TCP) calibration. The authors propose a new solution which employs a full camera model positioned on the effector of an industrial robotic arm. The proposed full camera model simulates the capture of a calibration pattern for use in automated TCP calibration. The study describes an experimental test robot stand for producing a reference data set, a full camera model, the parameters of a generally known camera obscura model, and a comparison of proposed solution with the camera obscura model. The results are discussed in the context of an innovative approach which features a full camera model to assist the TCP calibration process. The results showed that the full camera model produced greater accuracy, a significant benefit not provided by other state-of-the-art methods. In several cases, the absolute error produced was up to seven times lower than with the state-of-the-art camera obscura model. The error for small rotation (max. of 5[Formula: see text]) and small translation (max. of 20 mm) was 3.65 pixels. The results also highlighted the applicability of the proposed solution in real-life industrial processes.
Collapse
Affiliation(s)
- Jaromir Konecny
- VSB - Technical University of Ostrava, 17. listopadu 2172/15, Ostrava, 708 00, Czech Republic.
| | - Petr Beremlijski
- VSB - Technical University of Ostrava, 17. listopadu 2172/15, Ostrava, 708 00, Czech Republic
| | - Michaela Bailova
- VSB - Technical University of Ostrava, 17. listopadu 2172/15, Ostrava, 708 00, Czech Republic
| | - Zdenek Machacek
- VSB - Technical University of Ostrava, 17. listopadu 2172/15, Ostrava, 708 00, Czech Republic
| | - Jiri Koziorek
- VSB - Technical University of Ostrava, 17. listopadu 2172/15, Ostrava, 708 00, Czech Republic
| | - Michal Prauzek
- VSB - Technical University of Ostrava, 17. listopadu 2172/15, Ostrava, 708 00, Czech Republic
| |
Collapse
|
4
|
Burger L, Sharan L, Karl R, Wang C, Karck M, De Simone R, Wolf I, Romano G, Engelhardt S. Comparative evaluation of three commercially available markerless depth sensors for close-range use in surgical simulation. Int J Comput Assist Radiol Surg 2023:10.1007/s11548-023-02887-1. [PMID: 37140737 DOI: 10.1007/s11548-023-02887-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/09/2023] [Accepted: 03/27/2023] [Indexed: 05/05/2023]
Abstract
PURPOSE Minimally invasive surgeries have restricted surgical ports, demanding a high skill level from the surgeon. Surgical simulation potentially reduces this steep learning curve and additionally provides quantitative feedback. Markerless depth sensors show great promise for quantification, but most such sensors are not designed for accurate reconstruction of complex anatomical forms in close-range. METHODS This work compares three commercially available depth sensors, namely the Intel D405, D415, and the Stereolabs Zed-Mini in the range of 12-20 cm, for use in surgical simulation. Three environments are designed that closely mimic surgical simulation, comprising planar surfaces, rigid objects, and mitral valve models of silicone and realistic porcine tissue. The cameras are evaluated on Z-accuracy, temporal noise, fill rate, checker distance, point cloud comparisons, and visual inspection of surgical scenes, across several camera settings. RESULTS The Intel cameras show sub-mm accuracy in most static environments. The D415 fails in reconstructing valve models, while the Zed-Mini provides lesser temporal noise and higher fill rate. The D405 could reconstruct anatomical structures like the mitral valve leaflet and a ring prosthesis, but performs poorly for reflective surfaces like surgical tools and thin structures like sutures. CONCLUSION If a high temporal resolution is needed and lower spatial resolution is acceptable, the Zed-Mini is the best choice, whereas the Intel D405 is the most suited for close-range applications. The D405 shows potential for applications like deformable registration of surfaces, but is not yet suitable for applications like real-time tool tracking or surgical skill assessment.
Collapse
Affiliation(s)
- Lukas Burger
- Department of Cardiac Surgery, Heidelberg University Hospital, Heidelberg, Germany
- Department of Computer Science, Mannheim University of Applied Sciences, Mannheim, Germany
| | - Lalith Sharan
- Department of Cardiac Surgery, Heidelberg University Hospital, Heidelberg, Germany.
- DZHK (German Centre for Cardiovascular Research), Heidelberg, Germany.
| | - Roger Karl
- Department of Cardiac Surgery, Heidelberg University Hospital, Heidelberg, Germany
- DZHK (German Centre for Cardiovascular Research), Heidelberg, Germany
| | - Christina Wang
- Department of Cardiac Surgery, Heidelberg University Hospital, Heidelberg, Germany
| | - Matthias Karck
- Department of Cardiac Surgery, Heidelberg University Hospital, Heidelberg, Germany
| | - Raffaele De Simone
- Department of Cardiac Surgery, Heidelberg University Hospital, Heidelberg, Germany
| | - Ivo Wolf
- Department of Computer Science, Mannheim University of Applied Sciences, Mannheim, Germany
| | - Gabriele Romano
- Department of Cardiac Surgery, Heidelberg University Hospital, Heidelberg, Germany
| | - Sandy Engelhardt
- Department of Cardiac Surgery, Heidelberg University Hospital, Heidelberg, Germany
- DZHK (German Centre for Cardiovascular Research), Heidelberg, Germany
| |
Collapse
|
5
|
A Novel Method for Fast Generation of 3D Objects from Multiple Depth Sensors. JOURNAL OF ARTIFICIAL INTELLIGENCE AND SOFT COMPUTING RESEARCH 2023. [DOI: 10.2478/jaiscr-2023-0009] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/18/2023] Open
Abstract
Abstract
Scanning real 3D objects face many technical challenges. Stationary solutions allow for accurate scanning. However, they usually require special and expensive equipment. Competitive mobile solutions (handheld scanners, LiDARs on vehicles, etc.) do not allow for an accurate and fast mapping of the surface of the scanned object. The article proposes an end-to-end automated solution that enables the use of widely available mobile and stationary scanners. The related system generates a full 3D model of the object based on multiple depth sensors. For this purpose, the scanned object is marked with markers. Markers type and positions are automatically detected and mapped to a template mesh. The reference template is automatically selected for the scanned object, which is then transformed according to the data from the scanners with non-rigid transformation. The solution allows for the fast scanning of complex and varied size objects, constituting a set of training data for segmentation and classification systems of 3D scenes. The main advantage of the proposed solution is its efficiency, which enables real-time scanning and the ability to generate a mesh with a regular structure. It is critical for training data for machine learning algorithms. The source code is available at https://github.com/SATOffice/improved_scanner3D.
Collapse
|
6
|
Jung S, Lee YS, Lee Y, Lee K. 3D Reconstruction Using 3D Registration-Based ToF-Stereo Fusion. SENSORS (BASEL, SWITZERLAND) 2022; 22:8369. [PMID: 36366067 PMCID: PMC9654747 DOI: 10.3390/s22218369] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 10/12/2022] [Revised: 10/26/2022] [Accepted: 10/27/2022] [Indexed: 06/16/2023]
Abstract
Depth sensing is an important issue in many applications, such as Augmented Reality (AR), eXtended Reality (XR), and Metaverse. For 3D reconstruction, a depth map can be acquired by a stereo camera and a Time-of-Flight (ToF) sensor. We used both sensors complementarily to improve the accuracy of 3D information of the data. First, we applied a generalized multi-camera calibration method that uses both color and depth information. Next, depth maps of two sensors were fused by 3D registration and reprojection approach. Then, hole-filling was applied to refine the new depth map from the ToF-stereo fused data. Finally, the surface reconstruction technique was used to generate mesh data from the ToF-stereo fused pointcloud data. The proposed procedure was implemented and tested with real-world data and compared with various algorithms to validate its efficiency.
Collapse
|
7
|
Ryselis K, Blažauskas T, Damaševičius R, Maskeliūnas R. Agrast-6: Abridged VGG-Based Reflected Lightweight Architecture for Binary Segmentation of Depth Images Captured by Kinect. SENSORS (BASEL, SWITZERLAND) 2022; 22:6354. [PMID: 36080813 PMCID: PMC9460068 DOI: 10.3390/s22176354] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 07/03/2022] [Revised: 08/15/2022] [Accepted: 08/19/2022] [Indexed: 06/15/2023]
Abstract
Binary object segmentation is a sub-area of semantic segmentation that could be used for a variety of applications. Semantic segmentation models could be applied to solve binary segmentation problems by introducing only two classes, but the models to solve this problem are more complex than actually required. This leads to very long training times, since there are usually tens of millions of parameters to learn in this category of convolutional neural networks (CNNs). This article introduces a novel abridged VGG-16 and SegNet-inspired reflected architecture adapted for binary segmentation tasks. The architecture has 27 times fewer parameters than SegNet but yields 86% segmentation cross-intersection accuracy and 93% binary accuracy. The proposed architecture is evaluated on a large dataset of depth images collected using the Kinect device, achieving an accuracy of 99.25% in human body shape segmentation and 87% in gender recognition tasks.
Collapse
|
8
|
ISVD-Based Advanced Simultaneous Localization and Mapping (SLAM) Algorithm for Mobile Robots. MACHINES 2022. [DOI: 10.3390/machines10070519] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/06/2023]
Abstract
In the case of simultaneous localization and mapping, route planning and navigation are based on data captured by multiple sensors, including built-in cameras. Nowadays, mobile devices frequently have more than one camera with overlapping fields of view, leading to solutions where depth information can also be gathered along with ordinary RGB color data. Using these RGB-D sensors, two- and three-dimensional point clouds can be recorded from the mobile devices, which provide additional information for localization and mapping. The method of matching point clouds during the movement of the device is essential: reducing noise while having an acceptable processing time is crucial for a real-life application. In this paper, we present a novel ISVD-based method for displacement estimation, using key points detected by SURF and ORB feature detectors. The ISVD algorithm is a fitting procedure based on SVD resolution, which removes outliers from the point clouds to be fitted in several steps. The developed method removes these outlying points in several steps, in each iteration examining the relative error of the point pairs and then progressively reducing the maximum error for the next matching step. An advantage over relevant methods is that this method always gives the same result, as no random steps are included.
Collapse
|