Video-based augmented reality combining CT-scan and instrument position data to microscope view in middle ear surgery

The aim of the study was to develop and assess the performance of a video-based augmented reality system, combining preoperative computed tomography (CT) and real-time microscopic video, as the first crucial step to keyhole middle ear procedures through a tympanic membrane puncture. Six different artificial human temporal bones were included in this prospective study. Six stainless steel fiducial markers were glued on the periphery of the eardrum, and a high-resolution CT-scan of the temporal bone was obtained. Virtual endoscopy of the middle ear based on this CT-scan was conducted on Osirix software. Virtual endoscopy image was registered to the microscope-based video of the intact tympanic membrane based on fiducial markers and a homography transformation was applied during microscope movements. These movements were tracked using Speeded-Up Robust Features (SURF) method. Simultaneously, a micro-surgical instrument was identified and tracked using a Kalman filter. The 3D position of the instrument was extracted by solving a three-point perspective framework. For evaluation, the instrument was introduced through the tympanic membrane and ink droplets were injected on three middle ear structures. An average initial registration accuracy of 0.21 ± 0.10 mm (n = 3) was achieved with a slow propagation error during tracking (0.04 ± 0.07 mm). The estimated surgical instrument tip position error was 0.33 ± 0.22 mm. The target structures’ localization accuracy was 0.52 ± 0.15 mm. The submillimetric accuracy of our system without tracker is compatible with ear surgery.

Simultaneous Multi-Structure Segmentation and 3D Nonrigid Pose Estimation in Image-Guided Robotic Surgery

In image-guided robotic surgery, segmenting the endoscopic video stream into meaningful parts provides important contextual information that surgeons can exploit to enhance their perception of the surgical scene. This information provides surgeons with real-time decision-making guidance before initiating critical tasks such as tissue cutting. Segmenting endoscopic video is a challenging problem due to a variety of complications including significant noise attributed to bleeding and smoke from cutting, poor appearance contrast between different tissue types, occluding surgical tools, and limited visibility of the objects' geometries on the projected camera views. In this paper, we propose a multi-modal approach to segmentation where preoperative 3D computed tomography scans and intraoperative stereo-endoscopic video data are jointly analyzed. The idea is to segment multiple poorly visible structures in the stereo/multichannel endoscopic videos by fusing reliable prior knowledge captured from the preoperative 3D scans. More specifically, we estimate and track the pose of the preoperative models in 3D and consider the models' non-rigid deformations to match with corresponding visual cues in multi-channel endoscopic video and segment the objects of interest. Further, contrary to most augmented reality frameworks in endoscopic surgery that assume known camera parameters, an assumption that is often violated during surgery due to non-optimal camera calibration and changes in camera focus/zoom, our method embeds these parameters into the optimization hence correcting the calibration parameters within the segmentation process. We evaluate our technique on synthetic data, ex vivo lamb kidney datasets, and in vivo clinical partial nephrectomy surgery with results demonstrating high accuracy and robustness.

Surgical planning of Isshiki type I thyroplasty using an open-source Digital Imaging and Communication in Medicine viewer OsiriX

CONCLUSION

Preoperative surgical planning of Isshiki type I thyroplasty with the Digital Imaging and Communication in Medicine (DICOM) viewer OsiriX can be used for strategic and predictable type I thyroplasty.

OBJECTIVES

The aim of this study was to determine the efficacy of the preoperative planning of type I thyroplasty using the DICOM viewer OsiriX.

METHODS

Five patients with unilateral vocal cord paralysis and severe breathy dysphonia were included in this study. Multidetector computed tomography (CT) DICOM data were obtained and breath holding was performed during image acquisition. Using three-dimensional multiplanar reconstruction, we outlined the optimal location for a window. Type I thyroplasty was performed using Isshiki's original method, and only the placement of the window was decided according to the preoperative simulation point. To verify the advantages of this method, we compared our data with the previous data for total operation time and voice quality at 3 months after the operation without the DICOM viewer planning.

RESULTS

All patients are free from dysphonia and their glottic closures have resolved satisfactorily. Postoperative CT revealed that appropriate implant positioning resulted in successful surgical intervention. The comparison of total operation time showed that with the new method, operation duration was shortened by 12 min.

Markerless tracking for augmented reality for image-guided Endoscopic Retrograde Cholangiopancreatography

This paper proposes a markerless tracking method with adaptive pose estimation for augmenting 3D organ models on top of the endoscopic image for Endoscopic Retrograde Cholangiopancreatography (ERCP). While many applications of augmented reality (AR) to surgeries need special markers to track the camera's position and orientation in the live video, our method employs the feature detection techniques to track the endoscopic camera. One of the most difficult problems when applying feature-based method to AR for ERCP is the lack of texture & highly specular reflection surface of duodenum in the endoscopic images, which does not provide a stable number of keypoints to track in the endoscopic video sequence. By introducing an adaptive weight function in the combination of reference-current frame tracking with previous-current frame tracking, we enhance the tracking performance remarkably. The proposed method is evaluated using an endoscopic video of a real ERCP and 3D duodenum model reconstructed from CT data of the patient. The result shows real-time performance and robustness of the method.

Interactive visualization and analysis of multimodal datasets for surgical applications

Surgeons use information from multiple sources when making surgical decisions. These include volumetric datasets (such as CT, PET, MRI, and their variants), 2D datasets (such as endoscopic videos), and vector-valued datasets (such as computer simulations). Presenting all the information to the user in an effective manner is a challenging problem. In this paper, we present a visualization approach that displays the information from various sources in a single coherent view. The system allows the user to explore and manipulate volumetric datasets, display analysis of dataset values in local regions, combine 2D and 3D imaging modalities and display results of vector-based computer simulations. Several interaction methods are discussed: in addition to traditional interfaces including mouse and trackers, gesture-based natural interaction methods are shown to control these visualizations with real-time performance. An example of a medical application (medialization laryngoplasty) is presented to demonstrate how the combination of different modalities can be used in a surgical setting with our approach.