Visual SLAM for Handheld Monocular Endoscope

Intraoperative on-the-fly organ-mosaicking for laparoscopic surgery

The goal of computer-assisted surgery is to provide the surgeon with guidance during an intervention, e.g., using augmented reality. To display preoperative data, soft tissue deformations that occur during surgery have to be taken into consideration. Laparoscopic sensors, such as stereo endoscopes, can be used to create a three-dimensional reconstruction of stereo frames for registration. Due to the small field of view and the homogeneous structure of tissue, reconstructing just one frame, in general, will not provide enough detail to register preoperative data, since every frame only contains a part of an organ surface. A correct assignment to the preoperative model is possible only if the patch geometry can be unambiguously matched to a part of the preoperative surface. We propose and evaluate a system that combines multiple smaller reconstructions from different viewpoints to segment and reconstruct a large model of an organ. Using graphics processing unit-based methods, we achieved four frames per second. We evaluated the system with in silico, phantom, ex vivo, and in vivo (porcine) data, using different methods for estimating the camera pose (optical tracking, iterative closest point, and a combination). The results indicate that the proposed method is promising for on-the-fly organ reconstruction and registration.

2-DOF auto-calibration for a 3D endoscope system based on active stereo

For endoscopic medical treatment, measuring the size and shape of lesions, such as tumors, is important. We are developing a 3D endoscope system to measure the shape and size of living tissues based on active stereo. In previous works, our group attached a pattern projector outside the endoscope head. Since this increased the diameter of the endoscope, the burden and the risks of the patients would increase. In this paper, we set the pattern projector inside the instrument channel of the endoscope instead of mounting it outside, so that it can be deployed whenever required. This does not increase the size of the endoscope and facilitates the measuring process. However, since the projector is not physically fixed to the endoscope anymore prior to the operation, we propose an "auto-calibration" technique where extrinsic parameters are calibrated intra-operatively from a point marker on the projector observed simultaneously on the target surface. In the experiment, we show that the external parameters were successfully calibrated to obtain 3D reconstructions properly with the overall systems. The accuracy of the auto-calibration was validated by confirming that the epipolar constraints were kept, and a 3D reconstruction of a human tissue was demonstrated.

Stitching and Surface Reconstruction From Endoscopic Image Sequences: A Review of Applications and Methods

Endoscopic procedures form part of routine clinical practice for minimally invasive examinations and interventions. While they are beneficial for the patient, reducing surgical trauma and making convalescence times shorter, they make orientation and manipulation more challenging for the physician, due to the limited field of view through the endoscope. However, this drawback can be reduced by means of medical image processing and computer vision, using image stitching and surface reconstruction methods to expand the field of view. This paper provides a comprehensive overview of the current state of the art in endoscopic image stitching and surface reconstruction. The literature in the relevant fields of application and algorithmic approaches is surveyed. The technological maturity of the methods and current challenges and trends are analyzed.

Hierarchical structure from motion optical flow algorithms to harvest three-dimensional features from two-dimensional neuro-endoscopic images

Technical advances have led to an increase in the use of the endoscope in neurosurgery in recent years, particularly for intraventricular procedures and pituitary and anterior skull base surgery. Recently stereoscopic three-dimensional (3D) endoscopes have become available and may over time replace traditional two-dimensional (2D) imagery. An alternative strategy would be to use computer software algorithms to give monocular 2D endoscopes 3D capabilities. In this study our objective was to recover depth information from 2D endoscopic images using optical flow techniques. Digital images were recorded using a 2D endoscope and a hierarchical structure from motion algorithm was applied to the motion of the endoscope in order to calculate depth information for the generation of 3D anatomical structure. We demonstrate that 3D data can be recovered from 2D endoscopic images taken during endoventricular surgery where there is a mix of rapid camera motion and periods where the camera is nearly stationary. These algorithms may have the potential to give 3D visualization capabilities to 2D endoscopic hardware.

Computer vision distance measurement from endoscopic sequences: prospective evaluation in laparoscopic ventral hernia repair

BACKGROUND

Research in computer vision and mobile robotics has developed a family of popular algorithms known as Visual Simultaneous Localization And Mapping (Visual SLAM). These algorithms can provide 3D models of body cavities using the images obtained from standard monocular endoscopes. The 3D models can be used to estimate hernia defect measurements during laparoscopic ventral hernia repair (LVHR).

METHODS

We conducted a descriptive and comparative prospective study to analyze results from 15 patients who underwent LVHR. Three methods of measurement were used in each patient: two classical methods (needle and tape) and a new visual SLAM measurement (VSM) method. The major and minor axes of the ellipse-shaped hernia defect were measured.

RESULTS

Both axes could be measured using the VSM method in all patients except one (93%). The tape method measured 63% of the axes, but was difficult to perform because of patient comorbidities and because of limited range of motion of the laparoscopic tools. The needle method obtained 73% of measurements, because of patient comorbidities. The tape method was the most accurate (accuracy up to 0.5 cm because of tape resolution). The needle method was relatively inaccurate, with a mean error of >3 cm. The VSM method was as accurate as the tape method. The mean time taken to perform measurements was 40 s for the VSM method (range 29-60 s), 169 s for the needle method (range 66-300 s), and 186 s for the tape method (range 110-322 s).

CONCLUSIONS

The needle method is relatively inaccurate and invasive. The tape method is accurate, but is not easy to perform and is relatively time consuming. The VSM method is noninvasive and fast and is as accurate as the tape method.