A High-Precision Hand-Eye Coordination Localization Method under Convex Relaxation Optimization

Traditional switching operations require on-site work, and the high voltage generated by arc discharges can pose a risk of injury to the operator. Therefore, a combination of visual servo and robot control is used to localize the switching operation and construct hand-eye calibration equations. The solution to the hand-eye calibration equations is coupled with the rotation matrix and translation vectors, and it depends on the initial value determination. This article presents a convex relaxation global optimization hand-eye calibration algorithm based on dual quaternions. Firstly, the problem model is simplified using the mathematical tools of dual quaternions, and then the linear matrix inequality convex optimization method is used to obtain a rotation matrix with higher accuracy. Afterwards, the calibration equations of the translation vectors are rewritten, and a new objective function is established to solve the coupling influence between them, maintaining positioning precision at approximately 2.9 mm. Considering the impact of noise on the calibration process, Gaussian noise is added to the solutions of the rotation matrix and translation vector to make the data more closely resemble the real scene in order to evaluate the performance of different hand-eye calibration algorithms. Eventually, an experiment comparing different hand-eye calibration methods proves that the proposed algorithm is better than other hand-eye calibration algorithms in terms of calibration accuracy, robustness to noise, and stability, satisfying the accuracy requirements of switching operations.

Reprojection Error Analysis and Algorithm Optimization of Hand-Eye Calibration for Manipulator System

The Euclidean distance error of calibration results cannot be calculated during the hand-eye calibration process of a manipulator because the true values of the hand-eye conversion matrix cannot be obtained. In this study, a new method for error analysis and algorithm optimization is presented. An error analysis of the method is carried out using a priori knowledge that the location of the augmented reality markers is fixed during the calibration process. The coordinates of the AR marker center point are reprojected onto the pixel coordinate system and then compared with the true pixel coordinates of the AR marker center point obtained by corner detection or manual labeling to obtain the Euclidean distance between the two coordinates as the basis for the error analysis. We then fine-tune the results of the hand-eye calibration algorithm to obtain the smallest reprojection error, thereby obtaining higher-precision calibration results. The experimental results show that, compared with the Tsai-Lenz algorithm, the optimized algorithm in this study reduces the average reprojection error by 44.43% and the average visual positioning error by 50.63%. Therefore, the proposed optimization method can significantly improve the accuracy of hand-eye calibration results.

Visual dexterity: In-hand reorientation of novel and complex object shapes

In-hand object reorientation is necessary for performing many dexterous manipulation tasks, such as tool use in less structured environments, which remain beyond the reach of current robots. Prior works built reorientation systems assuming one or many of the following conditions: reorienting only specific objects with simple shapes, limited range of reorientation, slow or quasi-static manipulation, simulation-only results, the need for specialized and costly sensor suites, and other constraints that make the system infeasible for real-world deployment. We present a general object reorientation controller that does not make these assumptions. It uses readings from a single commodity depth camera to dynamically reorient complex and new object shapes by any rotation in real time, with the median reorientation time being close to 7 seconds. The controller was trained using reinforcement learning in simulation and evaluated in the real world on new object shapes not used for training, including the most challenging scenario of reorienting objects held in the air by a downward-facing hand that must counteract gravity during reorientation. Our hardware platform only used open-source components that cost less than 5000 dollars. Although we demonstrate the ability to overcome assumptions in prior work, there is ample scope for improving absolute performance. For instance, the challenging duck-shaped object not used for training was dropped in 56% of the trials. When it was not dropped, our controller reoriented the object within 0.4 radians (23°) 75% of the time.

Role of Reference Frames for a Safe Human-Robot Interaction

Safety plays a key role in human-robot interactions in collaborative robot (cobot) applications. This paper provides a general procedure to guarantee safe workstations allowing human operations, robot contributions, the dynamical environment, and time-variant objects in a set of collaborative robotic tasks. The proposed methodology focuses on the contribution and the mapping of reference frames. Multiple reference frame representation agents are defined at the same time by considering egocentric, allocentric, and route-centric perspectives. The agents are processed to provide a minimal and effective assessment of the ongoing human-robot interactions. The proposed formulation is based on the generalization and proper synthesis of multiple cooperating reference frame agents at the same time. Accordingly, it is possible to achieve a real-time assessment of the safety-related implications through the implementation and fast calculation of proper safety-related quantitative indices. This allows us to define and promptly regulate the controlling parameters of the involved cobot without velocity limitations that are recognized as the main disadvantage. A set of experiments has been realized and investigated to demonstrate the feasibility and effectiveness of the research by using a seven-DOF anthropomorphic arm in combination with a psychometric test. The acquired results agree with the current literature in terms of the kinematic, position, and velocity aspects; use measurement methods based on tests provided to the operator; and introduce novel features of work cell arranging, including the use of virtual instrumentation. Finally, the associated analytical-topological treatments have enabled the development of a safe and comfortable measure to the human-robot relation with satisfactory experimental results compared to previous research. Nevertheless, the robot posture, human perception, and learning technologies would have to apply research from multidisciplinary fields such as psychology, gesture, communication, and social sciences in order to be prepared for positioning in real-world applications that offer new challenges for cobot applications.

Weak dual generalized inverse of a dual matrix and its applications

Recently, the dual Moore-Penrose generalized inverse has been applied to study the linear dual equation when the dual Moore-Penrose generalized inverse of the coefficient matrix of the linear dual equation exists. Nevertheless, the dual Moore-Penrose generalized inverse exists only in partially dual matrices. In this paper, to study more general linear dual equation, we introduce the weak dual generalized inverse described by four dual equations, and is a dual Moore-Penrose generalized inverses for it when the latter exists. Any dual matrix has the weak dual generalized inverse and is unique. We obtain some basic properties and characterizations of the weak dual generalized inverse. Also, we investigate relationships among the weak dual generalized inverse, the Moore-Penrose dual generalized inverses and the dual Moore-Penrose generalized inverses, give the equivalent characterization and use some numerical examples to show that the three are different dual generalized inverse. Afterwards, by applying the weak dual generalized inverse we solve two special linear dual equations, one of which is consistent and the other is inconsistent. Neither of the coefficient matrices of the above two linear dual equations has dual Moore-Penrose generalized inverses.

Convolutional neural network-based pose mapping estimation as an alternative to traditional hand-eye calibration

The vision system is a crucial technology for realizing the automation and intelligence of industrial robots, and the accuracy of hand-eye calibration is crucial in determining the relationship between the camera and robot end. Parallel robots are widely used in automated assembly due to their high positioning accuracy and large carrying capacity, but traditional hand-eye calibration methods may not be applicable due to their limited motion range and resulting accuracy problems. To address this issue, we propose using a pose, nonlinear mapping estimation method to solve the hand-eye calibration problem and have constructed a 1-D pose estimation convolutional neural network (PECNN) with excellent performance, through experiments and discussions. The PECNN achieves an end-to-end mapping of the variation of the target object pose to the variation of the robot end pose. Our experiments have shown that the proposed hand-eye calibration method has high accuracy and can be applied to the automated assembly tasks of vision-guided parallel robots. Moreover, the method is also applicable to most parallel robots and tandem robots.

Calibration method of spatial transformations between the non-orthogonal two-axis turntable and its mounted camera

In order to expand the field of view and measuremenst range, the camera is often mounted on a two-axis turntable to perform various visual tasks. And the calibration of the position and attitude relationship between the mounted camera and the two-axis turntable is a prerequisite for visual measurement. The turntable is considered an ideal orthogonal two-axis turntable in conventional methods. However, the rotation axes of the actual two-axis turntable may be neither vertical nor intersecting, and the optical center of the mounted camera is not always located in the rotation center of the turntable even for orthogonal two-axis turntables. The quite difference between the actual physical model of the two-axis turntable and the ideal model can cause large errors. Therefore, what we believe to be a novel position and attitude calibration method between a non-orthogonal two-axis turntable and the mounted camera is proposed. This method describes the spatial hetero-planar lines relationship between the azimuth axis and pitch axis of the turntable accurately. By the geometric invariant characteristics of the mounted camera in motion, the axes of turntable are recovered and the base coordinate system is established, and the position and attitude of the camera are calibrated. Simulation and experiments verify the correctness and effectiveness of our proposed method.

An X-Ray C-Arm Guided Automatic Targeting System for Histotripsy

OBJECTIVE

Histotripsy is an emerging noninvasive, nonionizing and nonthermal focal cancer therapy that is highly precise and can create a treatment zone of virtually any size and shape. Current histotripsy systems rely on ultrasound imaging to target lesions. However, deep or isoechoic targets obstructed by bowel gas or bone can often not be treated safely using ultrasound imaging alone. This work presents an alternative x-ray C-arm based targeting approach and a fully automated robotic targeting system.

METHODS

The approach uses conventional cone beam CT (CBCT) images to localize the target lesion and 2D fluoroscopy to determine the 3D position and orientation of the histotripsy transducer relative to the C-arm. The proposed pose estimation uses a digital model and deep learning-based feature segmentation to estimate the transducer focal point relative to the CBCT coordinate system. Additionally, the integrated robotic arm was calibrated to the C-arm by estimating the transducer pose for four preprogrammed transducer orientations and positions. The calibrated system can then automatically position the transducer such that the focal point aligns with any target selected in a CBCT image.

RESULTS

The accuracy of the proposed targeting approach was evaluated in phantom studies, where the selected target location was compared to the center of the spherical ablation zones in post-treatment CBCTs. The mean and standard deviation of the Euclidean distance was 1.4 ±0.5 mm. The mean absolute error of the predicted treatment radius was 0.5 ±0.5 mm.

CONCLUSION

CBCT-based histotripsy targeting enables accurate and fully automated treatment without ultrasound guidance.

SIGNIFICANCE

The proposed approach could considerably decrease operator dependency and enable treatment of tumors not visible under ultrasound.

A novel hand-eye calibration method of picking robot based on TOF camera

Aiming at the stability of hand-eye calibration in fruit picking scene, a simple hand-eye calibration method for picking robot based on optimization combined with TOF (Time of Flight) camera is proposed. This method needs to fix the TOF depth camera at actual and calculated coordinates of the peach the end of the robot, operate the robot to take pictures of the calibration board from different poses, and record the current photographing poses to ensure that each group of pictures is clear and complete, so as to use the TOF depth camera to image the calibration board. Obtain multiple sets of calibration board depth maps and corresponding point cloud data, that is, "eye" data. Through the circle center extraction and positioning algorithm, the circle center points on each group of calibration plates are extracted, and a circle center sorting method based on the vector angle and the center of mass coordinates is designed to solve the circle center caused by factors such as mirror distortion, uneven illumination and different photographing poses. And through the tool center point of the actuator, the coordinate value of the circle center point on the four corners of each group of calibration plates in the robot end coordinate system is located in turn, and the "hand" data is obtained. Combined with the SVD method, And according to the obtained point residuals, the weight coefficients of the marker points are redistributed, and the hand-eye parameters are iteratively optimized, which improves the accuracy and stability of the hand-eye calibration. the method proposed in this paper has a better ability to locate the gross error under the environment of large gross errors. In order to verify the feasibility of the hand-eye calibration method, the indoor picking experiment was simulated, and the peaches were identified and positioned by combining deep learning and 3D vision to verify the proposed hand-eye calibration method. The JAKA six-axis robot and TuYang depth camera are used to build the experimental platform. The experimental results show that the method is simple to operate, has good stability, and the calibration plate is easy to manufacture and low in cost. work accuracy requirements.

A Robust Planar Marker-Based Visual SLAM

Many visual SLAM systems are generally solved using natural landmarks or optical flow. However, due to textureless areas, illumination change or motion blur, they often acquire poor camera poses or even fail to track. Additionally, they cannot obtain camera poses with a metric scale in the monocular case. In some cases (such as when calibrating the extrinsic parameters of camera-IMU), we prefer to sacrifice the flexibility of such methods to improve accuracy and robustness by using artificial landmarks. This paper proposes enhancements to the traditional SPM-SLAM, which is a system that aims to build a map of markers and simultaneously localize the camera pose. By placing the markers in the surrounding environment, the system can run stably and obtain accurate camera poses. To improve robustness and accuracy in the case of rotational movements, we improve the initialization, keyframes insertion and relocalization. Additionally, we propose a novel method to estimate marker poses from a set of images to solve the problem of planar-marker pose ambiguity. Compared with the state-of-art, the experiments show that our system achieves better accuracy in most public sequences and is more robust than SPM-SLAM under rotational movements. Finally, the open-source code is publicly available and can be found at GitHub.

A Novel Point Set Registration-Based Hand-Eye Calibration Method for Robot-Assisted Surgery

Pedicle screw insertion with robot assistance dramatically improves surgical accuracy and safety when compared with manual implantation. In developing such a system, hand-eye calibration is an essential component that aims to determine the transformation between a position tracking and robot-arm systems. In this paper, we propose an effective hand-eye calibration method, namely registration-based hand-eye calibration (RHC), which estimates the calibration transformation via point set registration without the need to solve the AX=XB equation. Our hand-eye calibration method consists of tool-tip pivot calibrations in two-coordinate systems, in addition to paired-point matching, where the point pairs are generated via the steady movement of the robot arm in space. After calibration, our system allows for robot-assisted, image-guided pedicle screw insertion. Comprehensive experiments are conducted to verify the efficacy of the proposed hand-eye calibration method. A mean distance deviation of 0.70 mm and a mean angular deviation of 0.68° are achieved by our system when the proposed hand-eye calibration method is used. Further experiments on drilling trajectories are conducted on plastic vertebrae as well as pig vertebrae. A mean distance deviation of 1.01 mm and a mean angular deviation of 1.11° are observed when the drilled trajectories are compared with the planned trajectories on the pig vertebrae.

No-code robotic programming for agile production: A new markerless-approach for multimodal natural interaction in a human-robot collaboration context

Industrial robots and cobots are widely deployed in most industrial sectors. However, robotic programming still needs a lot of time and effort in small batch sizes, and it demands specific expertise and special training, especially when various robotic platforms are required. Actual low-code or no-code robotic programming solutions are exorbitant and meager. This work proposes a novel approach for no-code robotic programming for end-users with adequate or no expertise in industrial robotic. The proposed method ensures intuitive and fast robotic programming by utilizing a finite state machine with three layers of natural interactions based on hand gesture, finger gesture, and voice recognition. The implemented system combines intelligent computer vision and voice control capabilities. Using a vision system, the human could transfer spatial information of a 3D point, lines, and trajectories using hand and finger gestures. The voice recognition system will assist the user in parametrizing robot parameters and interacting with the robot’s state machine. Furthermore, the proposed method will be validated and compared with state-of-the-art “Hand-Guiding” cobot devices within real-world experiments. The results obtained are auspicious, and indicate the capability of this novel approach for real-world deployment in an industrial context.

Significance of Camera Pixel Error in the Calibration Process of a Robotic Vision System

Although robotic vision systems offer a promising technology solution for rapid and reconfigurable in-process 3D inspection of complex and large parts in contemporary manufacturing, measurement accuracy poses a challenge for its wide deployment. One of the key issues in adopting a robotic vision system is to understand the extent of its measurement errors which are directly correlated with the calibration process. In this paper, a possible source of practical and inherent measurement uncertainties involved in the calibration process of a robotic vision system are discussed. The system considered in this work consists of an image sensor mounted on an industrial robot manipulator with six degrees of freedom. Based on a series of experimental tests and computer simulations, the paper gives a comprehensive performance comparison of different calibration approaches and shows the impact of measurement uncertainties on the calibration process. It has been found from the error sensitivity analysis that minor uncertainties in the calibration process can significantly affect the accuracy of the robotic vision system. Further investigations suggest that inducing errors in image calibration patterns can have an adverse effect on the hand–eye calibration process compared to the angular errors in the robot joints.

Research of Online Hand-Eye Calibration Method Based on ChArUco Board

To solve the problem of inflexibility of offline hand-eye calibration in "eye-in-hand" modes, an online hand-eye calibration method based on the ChArUco board is proposed in this paper. Firstly, a hand-eye calibration model based on the ChArUco board is established, by analyzing the mathematical model of hand-eye calibration, and the image features of the ChArUco board. According to the advantages of the ChArUco board, with both the checkerboard and the ArUco marker, an online hand-eye calibration algorithm based on the ChArUco board is designed. Then, the online hand-eye calibration algorithm, based on the ChArUco board, is used to realize the dynamic adjustment of the hand-eye position relationship. Finally, the hand-eye calibration experiment is carried out to verify the accuracy of the hand-eye calibration based on the ChArUco board. The robustness and accuracy of the proposed method are verified by online hand-eye calibration experiments. The experimental results show that the accuracy of the online hand-eye calibration method proposed in this paper is between 0.4 mm and 0.6 mm, which is almost the same as the offline hand-eye calibration accuracy. The method in this paper utilizes the advantages of the ChArUco board to realize online hand-eye calibration, which improves the flexibility and robustness of hand-eye calibration.

SE(n)++: An Efficient Solution to Multiple Pose Estimation Problems

In robotic applications, many pose problems involve solving the homogeneous transformation based on the special Euclidean group SE(n) . However, due to the nonconvexity of SE(n) , many of these solvers treat rotation and translation separately, and the computational efficiency is still unsatisfactory. A new technique called the SE(n)++ is proposed in this article that exploits a novel mapping from SE(n) to SO(n + 1) . The mapping transforms the coupling between rotation and translation into a unified formulation on the Lie group and gives better analytical results and computational performances. Specifically, three major pose problems are considered in this article, that is, the point-cloud registration, the hand-eye calibration, and the SE(n) synchronization. Experimental validations have confirmed the effectiveness of the proposed SE(n)++ method in open datasets.

Simultaneous Calibration of the Hand-Eye, Flange-Tool and Robot-Robot Relationship in Dual-Robot Collaboration Systems

A multi-robot collaboration system can complete more complex tasks than a single robot system. Ensuring the calibration accuracy between robots in the system is a prerequisite for the effective inter-robot cooperation. This paper presents a dual-robot system for orthopedic surgeries, where the relationships between hand-eye, flange-tool, and robot-robot need to be calibrated. This calibration problem can be summarized to the solution of the matrix equation of AXB=YCZ. A combined solution is proposed to solve the unknown parameters in the equation of AXB=YCZ, which consists of the dual quaternion closed-form method and the iterative method based on Levenberg-Marquardt (LM) algorithm. The closed-form method is used to quickly obtain the initial value for the iterative method so as to increase the convergence speed and calibration accuracy of the iterative method. Simulation and experimental analyses are carried out to verify the accuracy and effectiveness of the proposed method.

Solpen: An Accurate 6-DOF Positioning Tool for Vision-Guided Robotics

A robot trajectory teaching system with a vision-based positioning pen, which we called Solpen, is developed to generate pose paths of six degrees of freedom (6-DoF) for vision-guided robotics applications such as welding, cutting, painting, or polishing, which can achieve a millimeter dynamic accuracy within a meter working distance from the camera. The system is simple and requires only a 2D camera and the printed ArUco markers which are hand-glued on 31 surfaces of the designed 3D-printed Solpen. Image processing techniques are implemented to remove noise and sharpen the edge of the ArUco images and also enhance the contrast of the ArUco edge intensity generated by the pyramid reconstruction. In addition, the least squares method is implemented to optimize parameters for the center pose of the truncated Icosahedron center, and the vector of the Solpen-tip. From dynamic experiments conducted with ChArUco board to verify exclusively the pen performance, the developed system is robust within its working range, and achieves a minimum axis-accuracy at approximately 0.8 mm.

Optimal robot-world and hand-eye calibration with rotation and translation coupling

A classic hand-eye system involves hand-eye calibration and robot-world and hand-eye calibration. Insofar as hand-eye calibration can solve only hand-eye transformation, this study aims to determine the robot-world and hand-eye transformations simultaneously based on the robot-world and hand-eye equation. According to whether the rotation part and the translation part of the equation are decoupled, the methods can be divided into separable solutions and simultaneous solutions. The separable solutions solve the rotation part before solving the translation part, so the estimated errors of the rotation will be transferred to the translation. In this study, a method was proposed for calculation with rotation and translation coupling; a closed-form solution based on Kronecker product and an iterative solution based on the Gauss–Newton algorithm were involved. The feasibility was further tested using simulated data and real data, and the superiority was verified by comparison with the results obtained by the available method. Finally, we improved a method that can solve the singularity problem caused by the parameterization of the rotation matrix, which can be widely used in the robot-world and hand-eye calibration. The results show that the prediction errors of rotation and translation based on the proposed method be reduced to $0.26^\circ$ and $1.67$ mm, respectively.

A Novel Distribution for Representation of 6D Pose Uncertainty

The 6D Pose estimation is a crux in many applications, such as visual perception, autonomous navigation, and spacecraft motion. For robotic grasping, the cluttered and self-occlusion scenarios bring new challenges to the this field. Currently, society uses CNNs to solve this problem. The CNN models will suffer high uncertainty caused by the environmental factors and the object itself. These models usually maintain a Gaussian distribution, which is not suitable for the underlying manifold structure of the pose. Many works decouple rotation from the translation and quantify rotational uncertainty. Only a few works pay attention to the uncertainty of the 6D pose. This work proposes a distribution that can capture the uncertainty of the 6D pose parameterized by the dual quaternions, meanwhile, the proposed distribution takes the periodic nature of the underlying structure into account. The presented results include the normalization constant computation and parameter estimation techniques of the distribution. This work shows the benefits of the proposed distribution, which provides a more realistic explanation for the uncertainty in the 6D pose and eliminates the drawback inherited from the planar rigid motion.

Towards Cooperative Perception Services for ITS: Digital Twin in the Automotive Edge Cloud

We demonstrate a working functional prototype of a cooperative perception system that maintains a real-time digital twin of the traffic environment, providing a more accurate and more reliable model than any of the participant subsystems—in this case, smart vehicles and infrastructure stations—would manage individually. The importance of such technology is that it can facilitate a spectrum of new derivative services, including cloud-assisted and cloud-controlled ADAS functions, dynamic map generation with analytics for traffic control and road infrastructure monitoring, a digital framework for operating vehicle testing grounds, logistics facilities, etc. In this paper, we constrain our discussion on the viability of the core concept and implement a system that provides a single service: the live visualization of our digital twin in a 3D simulation, which instantly and reliably matches the state of the real-world environment and showcases the advantages of real-time fusion of sensory data from various traffic participants. We envision this prototype system as part of a larger network of local information processing and integration nodes, i.e., the logically centralized digital twin is maintained in a physically distributed edge cloud.

A Closed-Form Solution to Linear Feature-Based Registration of LiDAR Point Clouds

Due to the high complexity of geo-spatial entities and the limited field of view of LiDAR equipment, pairwise registration is a necessary step for integrating point clouds from neighbouring LiDAR stations. Considering that accurate extraction of point features is often difficult without the use of man-made reflectors, and the initial approximate values for the unknown transformation parameters must be estimated in advance to ensure the correct operation of those iterative methods, a closed-form solution to linear feature-based registration of point clouds is proposed in this study. Plücker coordinates are used to represent the linear features in three-dimensional space, whereas dual quaternions are employed to represent the spatial transformation. Based on the theory of least squares, an error norm (objective function) is first constructed by assuming that each pair of corresponding linear features is equivalent after registration. Then, by applying the extreme value analysis to the objective function, detailed derivations of the closed-form solution to the proposed linear feature-based registration method are given step by step. Finally, experimental tests are conducted on a real dataset. The derived experimental result demonstrates the feasibility of the proposed solution: By using eigenvalue decomposition to replace the linearization of the objective function, the proposed solution does not require any initial estimates of the unknown transformation parameters, which assures the stability of the registration method.

A Camera-Based Position Correction System for Autonomous Production Line Inspection

Visual inspection is an important task in manufacturing industries in order to evaluate the completeness and quality of manufactured products. An autonomous robot-guided inspection system was recently developed based on an offline programming (OLP) and RGB-D model system. This system allows a non-expert automatic optical inspection (AOI) engineer to easily perform inspections using scanned data. However, if there is a positioning error due to displacement or rotation of the object, this system cannot be used on a production line. In this study, we developed an automated position correction module to locate an object’s position and correct the robot’s pose and position based on the detected error values in terms of displacement or rotation. The proposed module comprised an automatic hand–eye calibration and the PnP algorithm. The automatic hand–eye calibration was performed using a calibration board to reduce manual error. After calibration, the PnP algorithm calculates the object position error using artificial marker images and compensates for the error to a new object on the production line. The position correction module then automatically maps the defined AOI target positions onto a new object, unless the target position changes. We performed experiments that showed that the robot-guided inspection system with the position correction module effectively performed the desired task. This smart innovative system provides a novel advancement by automating the AOI process on a production line to increase productivity.

Calibrating 3D Scanner in the Coordinate System of Optical Tracker for Image-To-Patient Registration

Three-dimensional scanners have been widely applied in image-guided surgery (IGS) given its potential to solve the image-to-patient registration problem. How to perform a reliable calibration between a 3D scanner and an external tracker is especially important for these applications. This study proposes a novel method for calibrating the extrinsic parameters of a 3D scanner in the coordinate system of an optical tracker. We bound an optical marker to a 3D scanner and designed a specified 3D benchmark for calibration. We then proposed a two-step calibration method based on the pointset registration technique and nonlinear optimization algorithm to obtain the extrinsic matrix of the 3D scanner. We applied repeat scan registration error (RSRE) as the cost function in the optimization process. Subsequently, we evaluated the performance of the proposed method on a recaptured verification dataset through RSRE and Chamfer distance (CD). In comparison with the calibration method based on 2D checkerboard, the proposed method achieved a lower RSRE (1.73 mm vs. 2.10, 1.94, and 1.83 mm) and CD (2.83 mm vs. 3.98, 3.46, and 3.17 mm). We also constructed a surgical navigation system to further explore the application of the tracked 3D scanner in image-to-patient registration. We conducted a phantom study to verify the accuracy of the proposed method and analyze the relationship between the calibration accuracy and the target registration error (TRE). The proposed scanner-based image-to-patient registration method was also compared with the fiducial-based method, and TRE and operation time (OT) were used to evaluate the registration results. The proposed registration method achieved an improved registration efficiency (50.72 ± 6.04 vs. 212.97 ± 15.91 s in the head phantom study). Although the TRE of the proposed registration method met the clinical requirements, its accuracy was lower than that of the fiducial-based registration method (1.79 ± 0.17 mm vs. 0.92 ± 0.16 mm in the head phantom study). We summarized and analyzed the limitations of the scanner-based image-to-patient registration method and discussed its possible development.

Hand-eye calibration method with a three-dimensional-vision sensor considering the rotation parameters of the robot pose

Hand-eye calibration is a fundamental step for a robot equipped with vision systems. However, this problem usually interacts with robot calibration because robot geometric parameters are not very precise. In this article, a new calibration method considering the rotation parameters of the robot pose is proposed. First, a constraint least square model is established assuming that each spherical center measurement of standard ball is equal in the robot base frame, which provides an initial solution. To further improve the solution accuracy, a nonlinear calibration model in the sensor frame is established. Since it can reduce one error accumulation process, a more accurate reference point can be used for optimization. Then, the rotation parameters of the robot pose whose slight errors cause large disturbance to the solution are selected by analyzing the coefficient matrices of the error items. Finally, the hand-eye transformation parameters are refined together with the rotation parameters in the nonlinear optimization solution. Some comparative simulations are performed between the modified least square method, constrained least square method, and the proposed method. The experiments are conducted on a 5-axis hybrid robot named TriMule to demonstrate the superior accuracy of the proposed method.

Camera-Robot Calibration for the da Vinci® Robotic Surgery System

The development of autonomous or semi-autonomous surgical robots stands to improve the performance of existing teleoperated equipment, but requires fine hand-eye calibration between the free-moving endoscopic camera and patient-side manipulator arms (PSMs). A novel method of solving this problem for the da Vinci® robotic surgical system and kinematically similar systems is presented. First, a series of image-processing and optical-tracking operations are performed to compute the coordinate transformation between the endoscopic camera view frame and an optical-tracking marker permanently affixed to the camera body. Then, the kinematic properties of the PSM are exploited to compute the coordinate transformation between the kinematic base frame of the PSM and an optical marker permanently affixed thereto. Using these transformations, it is then possible to compute the spatial relationship between the PSM and the endoscopic camera using only one tracker snapshot of the two markers. The effectiveness of this calibration is demonstrated by successfully guiding the PSM end effector to points of interest identified through the camera. Additional tests on a surgical task, namely grasping a surgical needle, are also performed to validate the proposed method. The resulting visually-guided robot positioning accuracy is better than the earlier hand-eye calibration results reported in the literature for the da Vinci® system, while supporting intraoperative update of the calibration and requiring only devices that are already commonly used in the surgical environment.

Vision-based hand-eye calibration for robot-assisted minimally invasive surgery

PURPOSE

The knowledge of laparoscope vision can greatly improve the surgical operation room (OR) efficiency. For the vision-based computer-assisted surgery, the hand-eye calibration establishes the coordinate relationship between laparoscope and robot slave arm. While significant advances have been made for hand-eye calibration in recent years, efficient algorithm for minimally invasive surgical robot is still a major challenge. Removing the external calibration object in abdominal environment to estimate the hand-eye transformation is still a critical problem.

METHODS

We propose a novel hand-eye calibration algorithm to tackle the problem which relies purely on surgical instrument already in the operating scenario for robot-assisted minimally invasive surgery (RMIS). Our model is formed by the geometry information of the surgical instrument and the remote center-of-motion (RCM) constraint. We also enhance the algorithm with stereo laparoscope model.

RESULTS

Promising validation of synthetic simulation and experimental surgical robot system have been conducted to evaluate the proposed method. We report results that the proposed method can exhibit the hand-eye calibration without calibration object.

CONCLUSION

Vision-based hand-eye calibration is developed. We demonstrate the feasibility to perform hand-eye calibration by taking advantage of the components of surgical robot system, leading to the efficiency of surgical OR.

Simultaneous Optimization of Patient-Image Registration and Hand-Eye Calibration for Accurate Augmented Reality in Surgery

OBJECTIVE

Augmented reality (AR) navigation using a position sensor in endoscopic surgeries relies on the quality of patient-image registration and hand-eye calibration. Conventional methods collect the necessary data to compute two output transformation matrices separately. However, the AR display setting during surgery generally differs from that during preoperative processes. Although conventional methods can identify optimal solutions under initial conditions, AR display errors are unavoidable during surgery owing to the inherent computational complexity of AR processes, such as error accumulation over successive matrix multiplications, and tracking errors of position sensor.

METHODS

We propose the simultaneous optimization of patient-image registration and hand-eye calibration in an AR environment before surgery. The relationship between the endoscope and a virtual object to overlay is first calculated using an endoscopic image, which also functions as a reference during optimization. After including the tracking information from the position sensor, patient-image registration and hand-eye calibration are optimized in terms of least-squares.

RESULTS

Experiments with synthetic data verify that the proposed method is less sensitive to computation and tracking errors. A phantom experiment with a position sensor is also conducted. The accuracy of the proposed method is significantly higher than that of the conventional method.

CONCLUSION

The AR accuracy of the proposed method is compared with those of the conventional ones, and the superiority of the proposed method is verified.

SIGNIFICANCE

This study demonstrates that the proposed method exhibits substantial potential for improving AR navigation accuracy.

Robust and Accurate Hand-Eye Calibration Method Based on Schur Matric Decomposition

To improve the accuracy and robustness of hand–eye calibration, a hand–eye calibration method based on Schur matric decomposition is proposed in this paper. The accuracy of these methods strongly depends on the quality of observation data. Therefore, preprocessing observation data is essential. As with traditional two-step hand–eye calibration methods, we first solve the rotation parameters and then the translation vector can be immediately determined. A general solution was obtained from one observation through Schur matric decomposition and then the degrees of freedom were decreased from three to two. Observation data preprocessing is one of the basic unresolved problems with hand–eye calibration methods. A discriminant equation to delete outliers was deduced based on Schur matric decomposition. Finally, the basic problem of observation data preprocessing was solved using outlier detection, which significantly improved robustness. The proposed method was validated by both simulations and experiments. The results show that the prediction error of rotation and translation was 0.06 arcmin and 1.01 mm respectively, and the proposed method performed much better in outlier detection. A minimal configuration for the unique solution was proven from a new perspective.

Bin-Picking for Planar Objects Based on a Deep Learning Network: A Case Study of USB Packs

Random bin-picking is a prominent, useful, and challenging industrial robotics application. However, many industrial and real-world objects are planar and have oriented surface points that are not sufficiently compact and discriminative for those methods using geometry information, especially depth discontinuities. This study solves the above-mentioned problems by proposing a novel and robust solution for random bin-picking for planar objects in a cluttered environment. Different from other research that has mainly focused on 3D information, this study first applies an instance segmentation-based deep learning approach using 2D image data for classifying and localizing the target object while generating a mask for each instance. The presented approach, moreover, serves as a pioneering method to extract 3D point cloud data based on 2D pixel values for building the appropriate coordinate system on the planar object plane. The experimental results showed that the proposed method reached an accuracy rate of 100% for classifying two-sided objects in the unseen dataset, and 3D appropriate pose prediction was highly effective, with average translation and rotation errors less than 0.23 cm and 2.26°, respectively. Finally, the system success rate for picking up objects was over 99% at an average processing time of 0.9 s per step, fast enough for continuous robotic operation without interruption. This showed a promising higher successful pickup rate compared to previous approaches to random bin-picking problems. Successful implementation of the proposed approach for USB packs provides a solid basis for other planar objects in a cluttered environment. With remarkable precision and efficiency, this study shows significant commercialization potential.

Calibration method for hand-eye system with rotation and translation couplings

This paper develops a novel hand-eye calibration method for hand-eye systems with rotation and translation coupling terms. First, a nonlinear camera model with distortion terms and a model of a hand-eye system with rotation and translation coupling terms are established. Based on a non-linear optimization method and a reverse projection method, a decoupling calibration method for a lower-degree-of-freedom hand-eye system is proposed. Then the path planning for the calibration process is carried out. Based on the analysis of coupling constraints and hand-eye system motion constraints, three types of hand-eye calibration paths with high efficiency and easy operation are developed. In addition, the influence of key parameters on hand-eye calibration accuracy is analyzed. Finally, calibration experiments and parametric influence experiments are carried out. The results demonstrate that the proposed method is effective and practical for calibrating the hand-eye system.

Batting an in-flight object to the target

Striking a flying object such as a ball to some target location is a highly skillful maneuver that a human being has to learn through a great deal of practice. In robotic manipulation, precision batting remains one of the most challenging tasks in which computer vision, modeling, planning, control, and action must be tightly coordinated in a split second. This paper investigates the problem of a two-degree-of-freedom robotic arm intercepting an object in free flight and redirecting it to some target with a single strike, assuming all the movements take place in one vertical plane. Two-dimensional impact is solved under Coulomb friction and energy-based restitution with a proof of termination. Planning combines impact dynamics and projectile flight mechanics with manipulator kinematics and image-based motion estimation. As the object is on the incoming flight, the post-impact task constraint of reaching the target is propagated backward in time, while the arm’s kinematic constraints are propagated forward (via joint trajectory interpolation), all to the pre-impact instant when they will meet constraints that allow batting to happen. All the constraints (16 in total) are then exerted on the arm’s pre-impact joint angles and velocities, which are repeatedly planned based on updated estimates of the object’s motion captured by a high-speed camera. The arm keeps adjusting its motion in sync with planning until batting takes place. Experiments have demonstrated a better batting performance by a Barrett Technology WAM Arm than by a human being without training.

Joint Stiffness Identification and Deformation Compensation of Serial Robots Based on Dual Quaternion Algebra

As the application of industrial robots is limited by low stiffness that causes low precision, a joint stiffness identification algorithm for serial robots is presented. In addition, a deformation compensation algorithm is proposed for the accuracy improvement. Both of these algorithms are formulated by dual quaternion algebra, which offers a compact, efficient, and singularity-free way in robot analysis. The joint stiffness identification algorithm is derived from stiffness modeling, which is the combination of the principle of virtual work and dual quaternion algebra. To validate the effectiveness of the proposed identification algorithm and deformation compensation algorithm, an experiment was conducted on a dual arm industrial robot SDA5F. The robot performed a drilling operation during the experiment, and the forces and torques that acted on the end-effector (EE) of both arms were measured in order to apply the deformation compensation algorithm. The results of the experiment show that the proposed identification algorithm is able to identify the joint stiffness parameters of serial industrial robots, and the deformation compensation algorithm can improve the accuracy of the position and orientation of the EE. Furthermore, the performance of the forces and torques that acted on the EE during the operation were improved as well.

Simultaneous Robot⁻World and Hand⁻Eye Calibration without a Calibration Object

An extended robot–world and hand–eye calibration method is proposed in this paper to evaluate the transformation relationship between the camera and robot device. This approach could be performed for mobile or medical robotics applications, where precise, expensive, or unsterile calibration objects, or enough movement space, cannot be made available at the work site. Firstly, a mathematical model is established to formulate the robot-gripper-to-camera rigid transformation and robot-base-to-world rigid transformation using the Kronecker product. Subsequently, a sparse bundle adjustment is introduced for the optimization of robot–world and hand–eye calibration, as well as reconstruction results. Finally, a validation experiment including two kinds of real data sets is designed to demonstrate the effectiveness and accuracy of the proposed approach. The translation relative error of rigid transformation is less than 8/10,000 by a Denso robot in a movement range of 1.3 m × 1.3 m × 1.2 m. The distance measurement mean error after three-dimensional reconstruction is 0.13 mm.

Non-orthogonal one-step calibration method for robotized transcranial magnetic stimulation

BACKGROUND

Robotized transcranial magnetic stimulation (TMS) combines the benefits of neuro-navigation with automation and provides a precision brain stimulation method. Since the coil will normally remain unmounted between different clinical uses, hand/eye calibration and coil calibration are required before each experiment. Today, these two steps are still separate: hand/eye calibration is performed using methods proposed by Tsai/Lenz or Floris Ernst, and then the coil calibration is carried out based on the traditional TMS experimental step. The process is complex and time-consuming, and traditional coil calibration using a handheld probe is susceptible to greater calibration error.

METHODS

A novel one-step calibration method has been developed to confirm hand/eye and coil calibration results by formulating a matrix equation system and estimating its solution. Hand/eye calibration and coil calibration are performed to confirm the pose relationships of the marker/end effector 'X', probe/end effector 'Y', and robot/world 'Z'. First, the coil is fixed on the end effector of the robot. During the one-step calibration process, a marker is mounted on the top of the coil and a calibration probe is fixed at the actual effective position of the coil. Next, the robot end effector is moved to a series of random positions 'A', the tracking data of marker 'B' and probe 'C' is obtained correspondingly. Then, a matrix equation system AX = ZB and AY = ZC can be acquired, and it is computed using a least-squares approach. Finally, the calibration probe is removed after calibration, while the marker remains fixed to the coil during the TMS experiment. The methods were evaluated based on simulation data and on experimental data from an optical tracking device. We compared our methods with two classical methods: the QR24 method proposed by Floris Ernst and the handheld coil calibration method.

RESULTS

The new methods outperform the QR24 method in the aspect of translational accuracy and performs similarly in the aspect of rotational accuracy, the total translational error decreased more than fifty percent. The new approach also outperforms traditional handheld coil calibration of navigated TMS systems, the total translational error decreased three- to fourfold, and the rotational error decreased six- to eightfold. Furthermore, the convergence speed is improved 16- to 27-fold for the new algorithms.

CONCLUSION

These results suggest that the new method can be used for hand/eye and coil calibration of a robotized TMS system. Two complex steps can be simplified using a least-squares approach.