Convolutional neural network-based pose mapping estimation as an alternative to traditional hand-eye calibration

The vision system is a crucial technology for realizing the automation and intelligence of industrial robots, and the accuracy of hand-eye calibration is crucial in determining the relationship between the camera and robot end. Parallel robots are widely used in automated assembly due to their high positioning accuracy and large carrying capacity, but traditional hand-eye calibration methods may not be applicable due to their limited motion range and resulting accuracy problems. To address this issue, we propose using a pose, nonlinear mapping estimation method to solve the hand-eye calibration problem and have constructed a 1-D pose estimation convolutional neural network (PECNN) with excellent performance, through experiments and discussions. The PECNN achieves an end-to-end mapping of the variation of the target object pose to the variation of the robot end pose. Our experiments have shown that the proposed hand-eye calibration method has high accuracy and can be applied to the automated assembly tasks of vision-guided parallel robots. Moreover, the method is also applicable to most parallel robots and tandem robots.

Augmented reality-based surgical guidance for wrist arthroscopy with bone-shift compensation

BACKGROUND AND OBJECTIVES

Intraoperative joint condition is different from preoperative CT/MR due to the motion applied during surgery, inducing an inaccurate approach to surgical targets. This study aims to provide real-time augmented reality (AR)-based surgical guidance for wrist arthroscopy based on a bone-shift model through an in vivo computed tomography (CT) study.

METHODS

To accurately visualize concealed wrist bones on the intra-articular arthroscopic image, we propose a surgical guidance system with a novel bone-shift compensation method using noninvasive fiducial markers. First, to measure the effect of traction during surgery, two noninvasive fiducial markers were attached before surgery. In addition, two virtual link models connecting the wrist bones were implemented. When wrist traction occurs during the operation, the movement of the fiducial marker is measured, and bone-shift compensation is applied to move the virtual links in the direction of the traction. The proposed bone-shift compensation method was verified with the in vivo CT data of 10 participants. Finally, to introduce AR, camera calibration for the arthroscope parameters was performed, and a patient-specific template was used for registration between the patient and the wrist bone model. As a result, a virtual bone model with three-dimensional information could be accurately projected on a two-dimensional arthroscopic image plane.

RESULTS

The proposed method was possible to estimate the position of wrist bone in the traction state with an accuracy of 1.4 mm margin. After bone-shift compensation was applied, the target point error was reduced by 33.6% in lunate, 63.3% in capitate, 55.0% in scaphoid, and 74.8% in trapezoid than those in preoperative wrist CT. In addition, a phantom experiment was introduced simulating the real surgical environment. AR display allowed to expand the field of view (FOV) of the arthroscope and helped in visualizing the anatomical structures around the bones.

CONCLUSIONS

This study demonstrated the successful handling of AR error caused by wrist traction using the proposed method. In addition, the method allowed accurate AR visualization of the concealed bones and expansion of the limited FOV of the arthroscope. The proposed bone-shift compensation can also be applied to other joints, such as the knees or shoulders, by representing their bone movements using corresponding virtual links. In addition, the movement of the joint skin during surgery can be measured using noninvasive fiducial markers in the same manner as that used for the wrist joint.

Generalized n-Dimensional Rigid Registration: Theory and Applications

The generalized rigid registration problem in high-dimensional Euclidean spaces is studied. The loss function is minimized with an equivalent error formulation by the Cayley formula. The closed-form linear least-square solution to such a problem is derived which generates the registration covariances, i.e., uncertainty information of rotation and translation, providing quite accurate probabilistic descriptions. Simulation results indicate the correctness of the proposed method and also present its efficiency on computation-time consumption, compared with previous algorithms using singular value decomposition (SVD) and linear matrix inequality (LMI). The proposed scheme is then applied to an interpolation problem on the special Euclidean group SE(n) with covariance-preserving functionality. Finally, experiments on covariance-aided Lidar mapping show practical superiority in robotic navigation.

Optimal robot-world and hand-eye calibration with rotation and translation coupling

A classic hand-eye system involves hand-eye calibration and robot-world and hand-eye calibration. Insofar as hand-eye calibration can solve only hand-eye transformation, this study aims to determine the robot-world and hand-eye transformations simultaneously based on the robot-world and hand-eye equation. According to whether the rotation part and the translation part of the equation are decoupled, the methods can be divided into separable solutions and simultaneous solutions. The separable solutions solve the rotation part before solving the translation part, so the estimated errors of the rotation will be transferred to the translation. In this study, a method was proposed for calculation with rotation and translation coupling; a closed-form solution based on Kronecker product and an iterative solution based on the Gauss–Newton algorithm were involved. The feasibility was further tested using simulated data and real data, and the superiority was verified by comparison with the results obtained by the available method. Finally, we improved a method that can solve the singularity problem caused by the parameterization of the rotation matrix, which can be widely used in the robot-world and hand-eye calibration. The results show that the prediction errors of rotation and translation based on the proposed method be reduced to $0.26^\circ$ and $1.67$ mm, respectively.

Vision-based hand-eye calibration for robot-assisted minimally invasive surgery

PURPOSE

The knowledge of laparoscope vision can greatly improve the surgical operation room (OR) efficiency. For the vision-based computer-assisted surgery, the hand-eye calibration establishes the coordinate relationship between laparoscope and robot slave arm. While significant advances have been made for hand-eye calibration in recent years, efficient algorithm for minimally invasive surgical robot is still a major challenge. Removing the external calibration object in abdominal environment to estimate the hand-eye transformation is still a critical problem.

METHODS

We propose a novel hand-eye calibration algorithm to tackle the problem which relies purely on surgical instrument already in the operating scenario for robot-assisted minimally invasive surgery (RMIS). Our model is formed by the geometry information of the surgical instrument and the remote center-of-motion (RCM) constraint. We also enhance the algorithm with stereo laparoscope model.

RESULTS

Promising validation of synthetic simulation and experimental surgical robot system have been conducted to evaluate the proposed method. We report results that the proposed method can exhibit the hand-eye calibration without calibration object.

CONCLUSION

Vision-based hand-eye calibration is developed. We demonstrate the feasibility to perform hand-eye calibration by taking advantage of the components of surgical robot system, leading to the efficiency of surgical OR.

Hand–eye calibration and grasping pose calculation with motion error compensation and vertical-component correction for 4-R(2-SS) parallel robot

Due to motion constraint of 4-R(2-SS) parallel robot, it is difficult to calculate the translation component of hand–eye calibration based on the existing model solving method accurately. Additionally, the camera calibration error, robot motion error, and invalid calibration motion poses make it difficult to achieve fast and accurate online hand–eye calibration. Therefore, we propose a hand–eye calibration method with motion error compensation and vertical-component correction for 4-R(2-SS) parallel robot by improving the existing eye-to-hand model and solving method. Firstly, the eye-to-hand model of single camera is improved and the robot motion error in the improved model is compensated to reduce the influence of camera calibration error and robot motion error on model accuracy. Secondly, the vertical-component of hand–eye calibration is corrected based on vertical constraint between calibration plate and end effector in parallel robot to calculate the pose and motion error in calibration of 4-R(2-SS) parallel robot accurately. Thirdly, the nontrivial solution constraint of eye-to-hand model is constructed and adopted to remove invalid calibration motion poses and plan calibration motion. Finally, the proposed method was verified by experiments with a fruit sorting system based on 4-R(2-SS) parallel robot. Compared with random motion, the existing model, and solving method, the average time of online calibration based on planned motion decreases by 29.773 s and the average error of calibration based on the improved model and solving method decreases by 151.293. The proposed method can improve the accuracy and efficiency of hand–eye calibration of 4-R(2-SS) parallel robot effectively and further realize accurate and fast grasping.

Simultaneous Optimization of Patient-Image Registration and Hand-Eye Calibration for Accurate Augmented Reality in Surgery

OBJECTIVE

Augmented reality (AR) navigation using a position sensor in endoscopic surgeries relies on the quality of patient-image registration and hand-eye calibration. Conventional methods collect the necessary data to compute two output transformation matrices separately. However, the AR display setting during surgery generally differs from that during preoperative processes. Although conventional methods can identify optimal solutions under initial conditions, AR display errors are unavoidable during surgery owing to the inherent computational complexity of AR processes, such as error accumulation over successive matrix multiplications, and tracking errors of position sensor.

METHODS

We propose the simultaneous optimization of patient-image registration and hand-eye calibration in an AR environment before surgery. The relationship between the endoscope and a virtual object to overlay is first calculated using an endoscopic image, which also functions as a reference during optimization. After including the tracking information from the position sensor, patient-image registration and hand-eye calibration are optimized in terms of least-squares.

RESULTS

Experiments with synthetic data verify that the proposed method is less sensitive to computation and tracking errors. A phantom experiment with a position sensor is also conducted. The accuracy of the proposed method is significantly higher than that of the conventional method.

CONCLUSION

The AR accuracy of the proposed method is compared with those of the conventional ones, and the superiority of the proposed method is verified.

SIGNIFICANCE

This study demonstrates that the proposed method exhibits substantial potential for improving AR navigation accuracy.

Simultaneous Robot⁻World and Hand⁻Eye Calibration without a Calibration Object

An extended robot–world and hand–eye calibration method is proposed in this paper to evaluate the transformation relationship between the camera and robot device. This approach could be performed for mobile or medical robotics applications, where precise, expensive, or unsterile calibration objects, or enough movement space, cannot be made available at the work site. Firstly, a mathematical model is established to formulate the robot-gripper-to-camera rigid transformation and robot-base-to-world rigid transformation using the Kronecker product. Subsequently, a sparse bundle adjustment is introduced for the optimization of robot–world and hand–eye calibration, as well as reconstruction results. Finally, a validation experiment including two kinds of real data sets is designed to demonstrate the effectiveness and accuracy of the proposed approach. The translation relative error of rigid transformation is less than 8/10,000 by a Denso robot in a movement range of 1.3 m × 1.3 m × 1.2 m. The distance measurement mean error after three-dimensional reconstruction is 0.13 mm.

Calibration of mobile manipulators using 2D positional features

Robotic manipulators are increasingly being attached to Automatic Ground Vehicles (AGVs) to aid in the efficiency of assembly for manufacturing systems. However, calibrating these mobile manipulators is difficult as the offset between the robotic manipulator and the AGV is often unknown. This paper provides a novel, simple, and low-cost method for calibrating and measuring the performance of mobile manipulators by using data collected from a laser retroreflector that digitally detects the horizontal two-dimensional (2D) position of reflectors on an artifact as well as a navigation system that provides the heading angle and 2D position of the AGV. The method is mathematically presented by providing a closed form solution to the positional component of the 2D robotworld/hand-eye calibration problem AX Y= B. The method is then applied to simulated data as well as data collected in a laboratory setting and compared to other methods.

Self-recalibration of a robot-assisted structured-light-based measurement system

The structured-light-based measurement method is widely employed in numerous fields. However, for industrial inspection, to achieve complete scanning of a work piece and overcome occlusion, the measurement system needs to be moved to different viewpoints. Moreover, frequent reconfiguration of the measurement system may be needed based on the size of the measured object, making the self-recalibration of extrinsic parameters indispensable. To this end, this paper proposes an automatic self-recalibration and reconstruction method, wherein a robot arm is employed to move the measurement system for complete scanning; the self-recalibration is achieved using fundamental matrix calculations and point cloud registration without the need for an accurate calibration gauge. Experimental results demonstrate the feasibility and accuracy of our method.

A new hand-eye calibration approach for fracture reduction robot

OBJECTIVE

The hand-eye calibration is used to determine the transformation between the end-effector and the camera marker of the robot. But the robot movement in traditional method would be time-consuming, inaccurate and even unavailable in some conditions. The method presented in this article can complete the calibration without any movement and is more suitable in clinical applications.

METHODS

Instead of solving the classic non-linear equation AX = XB, we collected the points on X and Y axes of the tool coordinate system (TCS) with the visual probe and fitted them using the singular value decomposition algorithm (SVD). Then, the transformation was obtained with the data of the tool center point (TCP). A comparison test was conducted to verify the performance of the method.

RESULTS

The average translation error and orientation error of the new method are 0.12 ± 0.122 mm and 0.18 ± 0.112° respectively, while they are 0.357 ± 0.347 mm and 0.416 ± 0.234° correspondingly in the traditional method.

CONCLUSIONS

The high accuracy of the method indicates that it is a good candidate for medical robots, which usually need to work in a sterile environment.

Visual Servoing from Lines

In this paper, we present a new approach to visual servoing using lines. It is based on a theoretical and geometrical study of the main line representations, which allows us to define a new representation, the so-called binormalized Plücker coordinates. These are particularly well suited for visual servoing. Indeed, they allow the definition of an image line alignment concept. Moreover, the control law which realizes such an alignment has several properties: partial decoupling between rotation and translation, analytical inversion of the motion equations and global asymptotic stability conditions. This control law was validated both in simulation and experimentally in the specific case of an orthogonal trihedron.

A METHOD FOR THE REGISTRATION OF A KNOWN CAD MODEL INTO THE WORKSPACE FRAME OF A ROBOT

In many robotic applications it is required to manipulate a specific rigid object whose CAD model is known a priori, but its position and orientation in space are unknown. This category of tasks includes piercing, painting or iron welding on some specific points of the rigid object's surface. For this kind of tasks to become feasible the CAD data of the rigid object should be registered into the robot's workspace frame, so that the robot arm becomes aware of the position and orientation of the rigid body with reference to its own flange or base coordinate frame. In order to achieve this goal several techniques from the fields of image processing, 3D modeling and robot kinematics should be combined. This paper provides a convenient combination of such methods that can be used for the successful on-line registration of a rigid object's CAD data to the robot's workspace frame.

A particle filtering framework for joint video tracking and pose estimation

A method is introduced to track the object's motion and estimate its pose directly from 2-D image sequences. Scale-invariant feature transform (SIFT) is used to extract corresponding feature points from image sequences. We demonstrate that pose estimation from the corresponding feature points can be formed as a solution to Sylvester's equation. We show that the proposed approach to the solution of Sylvester's equation is equivalent to the classical SVD method for 3D-3D pose estimation. However, whereas classical SVD cannot be used for pose estimation directly from 2-D image sequences, our method based on Sylvester's equation provides a new approach to pose estimation. Smooth video tracking and pose estimation is finally obtained by using the solution to Sylvester's equation within the importance sampling density of the particle filtering framework. Finally, computer simulation experiments conducted over synthetic data and real-world videos demonstrate the effectiveness of our method in both robustness and speed compared with other similar object tracking and pose estimation methods.

Data Selection for Hand-eye Calibration: A Vector Quantization Approach

This paper presents new vector quantization based methods for selecting well-suited data for hand-eye calibration from a given sequence of hand and eye movements. Data selection can improve the accuracy of classic hand-eye calibration, and make it possible in the first place in situations where the standard approach of manually selecting positions is inconvenient or even impossible, especially when using continuously recorded data. A variety of methods is proposed, which differ from each other in the dimensionality of the vector quantization compared to the degrees of freedom of the rotation representation, and how the rotation angle is incorporated. The performance of the proposed vector quantization based data selection methods is evaluated using data obtained from a manually moved optical tracking system (hand) and an endoscopic camera (eye).

Adaptive motion selection for online hand–eye calibration

While a robot moves, online hand–eye calibration to determine the relative pose between the robot gripper/end-effector and the sensors mounted on it is very important in a vision-guided robot system. During online hand–eye calibration, it is impossible to perform motion planning to avoid degenerate motions and small rotations, which may lead to unreliable calibration results. This paper proposes an adaptive motion selection algorithm for online hand–eye calibration, featured by dynamic threshold determination for motion selection and getting reliable hand–eye calibration results. Simulation and real experiments demonstrate the effectiveness of our method.

A hand-eye robotic model for total knee replacement surgery

This paper presents a hand-eye robotic model for total knee replacement (TKR) surgery. Unlike existent robot assisted TKR surgery, the proposed model is a surgical robot that combines with a movable hand-eye navigation system, which would use the full potential of both computer-assisted systems. Without using CT images and landmark pins in the patient's bones, it can directly measure the mechanical axis with high precision. This system provides a new approach of the minimally invasive surgery. Experiment results show that the proposed model is promising in the future application.

Real-Time Quality Control of Tracked Ultrasound

The overwhelming majority of intra-operative hazard situations in tracked ultrasound (US) systems are attributed to failure of registration between tracking and imaging coordinate frames. We introduce a novel methodology for eal-time in-vivo quality control of tracked US systems, in order to capture registration failures during the clinical procedure. In effect, we dynamically recalibrate the tracked US system for rotation, scale factor, and in-plane position offset up to a scale factor. We detect any unexpected change in these parameters through capturing discrepancies in the resulting calibration matrix, thereby assuring quality (accuracy and consistency) of the tracked system. No phantom is used for the recalibration. We perform the task of quality control in the background, transparently to the clinical user while the subject is being scanned. We present the concept, mathematical formulation, and experimental evaluation in-vitro. This new method can play an important role in guaranteeing accurate, consistent, and reliable performance of tracked ultrasound.