Development of a camera model and calibration procedure for oblique-viewing endoscopes

An improved camera model for oblique-viewing laparoscopes: high reprojection accuracy independent of telescope rotation

Objective. Oblique-viewing laparoscopes are popular in laparoscopic surgeries where the target anatomy is located in narrow areas. Their viewing direction can be shifted by telescope rotation without changing the laparoscope pose. This rotation also changes laparoscope camera parameters that are estimated by camera calibration to be able to reproject an anatomical model onto the laparoscopic view, creating augmented reality (AR). The aim of this study was to develop a camera model that accounts for these changes, achieving high reprojection accuracy for any telescope rotation.Approach. Camera parameters were acquired by calibrations encompassing a wide telescope rotation range. For those parameters showing periodic changes upon rotation, interpolation models were created and used to establish an updatable camera model. With this model, corner points of a tracked checkerboard were reprojected onto the checkerboard laparoscopic images, at random rotation angles. Root-mean-square reprojection errors (RMSEs) were calculated between the reprojected and imaged corner points.Main results. Reprojection RMSEs were low and approximately independent on telescope rotation angle, over a wide rotation range of 320°. The mean reprojection RMSE was 2.8±0.7 pixels for a conventional laparoscope and 3.6±0.7 pixels for a chip-on-the-tip (COTT) laparoscope, corresponding to 0.3±0.1 mm and 0.4±0.1 mm in world coordinates respectively. Worst-case reprojection errors were about 9 pixels (0.8 mm) for both laparoscopes.Significance. The camera model developed in this study improves on existing models for oblique-viewing laparoscopes because it provides high reprojection accuracy independent of the telescope rotation angle and is applicable for conventional and chip-on-a-tip oblique-viewing laparoscopes. The work presented here is an important step towards creating accurate AR in image-guided interventions where oblique-viewing laparoscopes are used while simultaneously providing the surgeon the flexibility to rotate the telescope to any desired rotation angle.Acronyms. CC: camera coordinates; CCToolbox: camera calibration toolbox; COTT: chip-on-the-tip; CS: camera sensor; DD: decentering distortion; FL: focal length; OTS: optical tracking system; PP: principal point; RD: radial distortion; SI: supplementary information;tHE:hand-eye translation component.

Hybrid electromagnetic-ArUco tracking of laparoscopic ultrasound transducer in laparoscopic video

Purpose: The purpose of this work was to develop a new method of tracking a laparoscopic ultrasound (LUS) transducer in laparoscopic video by combining the hardware [e.g., electromagnetic (EM)] and the computer vision-based (e.g., ArUco) tracking methods. Approach: We developed a special tracking mount for the imaging tip of the LUS transducer. The mount incorporated an EM sensor and an ArUco pattern registered to it. The hybrid method used ArUco tracking for ArUco-success frames (i.e., frames where ArUco succeeds in detecting the pattern) and used corrected EM tracking for the ArUco-failure frames. The corrected EM tracking result was obtained by applying correction matrices to the original EM tracking result. The correction matrices were calculated in previous ArUco-success frames by comparing the ArUco result and the original EM tracking result. Results: We performed phantom and animal studies to evaluate the performance of our hybrid tracking method. The corrected EM tracking results showed significant improvements over the original EM tracking results. In the animal study, 59.2% frames were ArUco-success frames. For the ArUco-failure frames, mean reprojection errors for the original EM tracking method and for the corrected EM tracking method were 30.8 pixel and 10.3 pixel, respectively. Conclusions: The new hybrid method is more reliable than using ArUco tracking alone and more accurate and practical than using EM tracking alone for tracking the LUS transducer in the laparoscope camera image. The proposed method has the potential to significantly improve tracking performance for LUS-based augmented reality applications.

Fast calibration of electromagnetically tracked oblique-viewing rigid endoscopes

PURPOSE

The oblique-viewing (i.e., angled) rigid endoscope is a commonly used tool in conventional endoscopic surgeries. The relative rotation between its two moveable parts, the telescope and the camera head, creates a rotation offset between the actual and the projection of an object in the camera image. A calibration method tailored to compensate such offset is needed.

METHODS

We developed a fast calibration method for oblique-viewing rigid endoscopes suitable for clinical use. In contrast to prior approaches based on optical tracking, we used electromagnetic (EM) tracking as the external tracking hardware to improve compactness and practicality. Two EM sensors were mounted on the telescope and the camera head, respectively, with considerations to minimize EM tracking errors. Single-image calibration was incorporated into the method, and a sterilizable plate, laser-marked with the calibration pattern, was also developed. Furthermore, we proposed a general algorithm to estimate the rotation center in the camera image. Formulas for updating the camera matrix in terms of clockwise and counterclockwise rotations were also developed.

RESULTS

The proposed calibration method was validated using a conventional [Formula: see text], 5-mm laparoscope. Freehand calibrations were performed using the proposed method, and the calibration time averaged 2 min and 8 s. The calibration accuracy was evaluated in a simulated clinical setting with several surgical tools present in the magnetic field of EM tracking. The root-mean-square re-projection error averaged 4.9 pixel (range 2.4-8.5 pixel, with image resolution of [Formula: see text] for rotation angles ranged from [Formula: see text] to [Formula: see text].

CONCLUSIONS

We developed a method for fast and accurate calibration of oblique-viewing rigid endoscopes. The method was also designed to be performed in the operating room and will therefore support clinical translation of many emerging endoscopic computer-assisted surgical systems.

Virtually transparent surgical instruments in endoscopic surgery with augmentation of obscured regions

PURPOSE

We developed and evaluated a visual compensation system that allows surgeons to visualize obscured regions in real time, such that the surgical instrument appears virtually transparent.

METHODS

The system consists of two endoscopes: a main endoscope to observe the surgical environment, and a supporting endoscope to render the region hidden from view by surgical instruments. The view captured by the supporting endoscope is transformed to simulate the view from the main endoscope, segmented to the shape of the hidden regions, and superimposed to the main endoscope image so that the surgical instruments look transparent. A prototype device was benchmarked for processing time and superimposition rendering error. Then, it was evaluated in a training environment with 22 participants performing a backhand needle driving task with needle exit point error as the criterion. Lastly, we conducted an in vivo study.

RESULTS

In the benchmark, the mean processing time was 62.4 ms, which was lower than the processing time accepted in remote surgeries. The mean superimposition error of the superimposed image was 1.4 mm. In the training environment, needle exit point error with the system decreased significantly for experts compared with the condition without the system. This change was not significant for novices. In the in vivo study, our prototype enabled visualization of needle exit points during anastomosis.

CONCLUSION

The benchmark suggests that the implemented system had an acceptable performance, and evaluation in the training environment demonstrated improved surgical task outcomes in expert surgeons. We will conduct a more comprehensive in vivo study in the future.