Soft tissue motion tracking with application to tablet-based incision planning in laser surgery

Surface deformation tracking in monocular laparoscopic video

Image-guided surgery has been proven to enhance the accuracy and safety of minimally invasive surgery (MIS). Nonrigid deformation tracking of soft tissue is one of the main challenges in image-guided MIS owing to the existence of tissue deformation, homogeneous texture, smoke and instrument occlusion, etc. In this paper, we proposed a piecewise affine deformation model-based nonrigid deformation tracking method. A Markov random field based mask generation method is developed to eliminate tracking anomalies. The deformation information vanishes when the regular constraint is invalid, which further deteriorates the tracking accuracy. Atime-series deformation solidification mechanism is introduced to reduce the degradation of the deformation field of the model. For the quantitative evaluation of the proposed method, we synthesized nine laparoscopic videos mimicking instrument occlusion and tissue deformation. Quantitative tracking robustness was evaluated on the synthetic videos. Three real videos of MIS containing challenges of large-scale deformation, large-range smoke, instrument occlusion, and permanent changes in soft tissue texture were also used to evaluate the performance of the proposed method. Experimental results indicate the proposed method outperforms state-of-the-art methods in terms of accuracy and robustness, which shows good performance in image-guided MIS.

μRALP and Beyond: Micro-Technologies and Systems for Robot-Assisted Endoscopic Laser Microsurgery

Laser microsurgery is the current gold standard surgical technique for the treatment of selected diseases in delicate organs such as the larynx. However, the operations require large surgical expertise and dexterity, and face significant limitations imposed by available technology, such as the requirement for direct line of sight to the surgical field, restricted access, and direct manual control of the surgical instruments. To change this status quo, the European project μRALP pioneered research towards a complete redesign of current laser microsurgery systems, focusing on the development of robotic micro-technologies to enable endoscopic operations. This has fostered awareness and interest in this field, which presents a unique set of needs, requirements and constraints, leading to research and technological developments beyond μRALP and its research consortium. This paper reviews the achievements and key contributions of such research, providing an overview of the current state of the art in robot-assisted endoscopic laser microsurgery. The primary target application considered is phonomicrosurgery, which is a representative use case involving highly challenging microsurgical techniques for the treatment of glottic diseases. The paper starts by presenting the motivations and rationale for endoscopic laser microsurgery, which leads to the introduction of robotics as an enabling technology for improved surgical field accessibility, visualization and management. Then, research goals, achievements, and current state of different technologies that can build-up to an effective robotic system for endoscopic laser microsurgery are presented. This includes research in micro-robotic laser steering, flexible robotic endoscopes, augmented imaging, assistive surgeon-robot interfaces, and cognitive surgical systems. Innovations in each of these areas are shown to provide sizable progress towards more precise, safer and higher quality endoscopic laser microsurgeries. Yet, major impact is really expected from the full integration of such individual contributions into a complete clinical surgical robotic system, as illustrated in the end of this paper with a description of preliminary cadaver trials conducted with the integrated μRALP system. Overall, the contribution of this paper lays in outlining the current state of the art and open challenges in the area of robot-assisted endoscopic laser microsurgery, which has important clinical applications even beyond laryngology.

Preclinical Performance Evaluation of a Robotic Endoscope for Non-Contact Laser Surgery

Despite great efforts, transoral robotic laser surgery has not been established clinically. Patient benefits are yet to be proven to accept shortcomings of robotic systems. In particular, laryngeal reachability and transition from microscope to accurate endoscopic laser ablation have not been achieved. We have addressed those challenges with a highly integrated robotic endoscope for non-contact endolaryngeal laser surgery. The current performance status has been assessed in multi-level user studies. In addition, the system was deployed to an ex vivo porcine larynx. The robotic design comprises an extensible continuum manipulator with multifunctional tip. The latter features laser optics, stereo vision, and illumination. Vision-based performance assessment is derived from depth estimation and scene tracking. Novices and experts (n = 20) conducted teleoperated delineation tasks to mimic laser ablation of delicate anatomy. Delineation with motion-compensated and raw endoscopic visualisation was carried out on planar and non-planar nominal patterns. Root mean square tracing errors of less than 0.75 mm were feasible with task completion times below 45 s. Relevant anatomy in the porcine larynx was exposed successfully. Accuracy and usability of the integrated platform bear potential for dexterous laser manipulation in clinical settings. Cadaver and in vivo animal studies may translate ex vivo findings.

Endoscopic vitreoretinal surgery: principles, applications and new directions

Purpose

To analyze endoscopic vitreoretinal surgery principles, applications, challenges and potential technological advances.

Background

Microendoscopic imaging permits vitreoretinal surgery for tissues that are not visible using operating microscopy ophthalmoscopy. Evolving instrumentation may overcome some limitations of current endoscopic technology.

Analysis

Transfer of the fine detail in endoscopic vitreoretinal images to extraocular video cameras is constrained currently by the caliber limitations of intraocular probes in ophthalmic surgery. Gradient index and Hopkins rod lenses provide high resolution ophthalmoscopy but restrict surgical manipulation. Fiberoptic coherent image guides offer surgical maneuverability but reduce imaging resolution. Coaxial endoscopic illumination can highlight delicate vitreoretinal structures difficult to image in chandelier or endoilluminator diffuse, side-scattered lighting. Microendoscopy’s ultra-high magnification video monitor images can reveal microscopic tissue details blurred partly by ocular media aberrations in contemporary surgical microscope ophthalmoscopy, thereby providing a lower resolution, invasive alternative to confocal fundus imaging. Endoscopic surgery is particularly useful when ocular media opacities or small pupils restrict or prevent transpupillary ophthalmoscopy. It has a growing spectrum of surgical uses that include the management of proliferative vitreoretinopathy and epiretinal membranes as well as the implantation of posterior chamber intraocular lenses and electrode arrays for intraretinal stimulation in retinitis pigmentosa. Microendoscopy’s range of applications will continue to grow with technological developments that include video microchip sensors, stereoscopic visualization, chromovitrectomy, digital image enhancement and operating room heads-up displays.

Conclusion

Microendoscopy is a robust platform for vitreoretinal surgery. Continuing clinical and technological innovation will help integrate it into the modern ophthalmic operating room of interconnected surgical microscopy, microendoscopy, vitrectomy machine and heads-up display instrumentation.

A dataset of laryngeal endoscopic images with comparative study on convolution neural network-based semantic segmentation

PURPOSE

Automated segmentation of anatomical structures in medical image analysis is a prerequisite for autonomous diagnosis as well as various computer- and robot-aided interventions. Recent methods based on deep convolutional neural networks (CNN) have outperformed former heuristic methods. However, those methods were primarily evaluated on rigid, real-world environments. In this study, existing segmentation methods were evaluated for their use on a new dataset of transoral endoscopic exploration.

METHODS

Four machine learning-based methods SegNet, UNet, ENet and ErfNet were trained with supervision on a novel 7-class dataset of the human larynx. The dataset contains 536 manually segmented images from two patients during laser incisions. The Intersection-over-Union (IoU) evaluation metric was used to measure the accuracy of each method. Data augmentation and network ensembling were employed to increase segmentation accuracy. Stochastic inference was used to show uncertainties of the individual models. Patient-to-patient transfer was investigated using patient-specific fine-tuning.

RESULTS

In this study, a weighted average ensemble network of UNet and ErfNet was best suited for the segmentation of laryngeal soft tissue with a mean IoU of 84.7%. The highest efficiency was achieved by ENet with a mean inference time of 9.22 ms per image. It is shown that 10 additional images from a new patient are sufficient for patient-specific fine-tuning.

CONCLUSION

CNN-based methods for semantic segmentation are applicable to endoscopic images of laryngeal soft tissue. The segmentation can be used for active constraints or to monitor morphological changes and autonomously detect pathologies. Further improvements could be achieved by using a larger dataset or training the models in a self-supervised manner on additional unlabeled data.

Volumetric 3D stitching of optical coherence tomography volumes

Optical coherence tomography (OCT) is a noninvasive medical imaging modality, which provides highresolution transectional images of biological tissue. However, its potential is limited due to a relatively small field of view. To overcome this drawback, we describe a scheme for fully automated stitching of multiple 3D OCT volumes for panoramic imaging. The voxel displacements between two adjacent images are calculated by extending the Lucas-Kanade optical flow a lgorithm to dense volumetric images. A RANSAC robust estimator is used to obtain rigid transformations out of the resulting flow v ectors. T he i mages a re t ransformed into the same coordinate frame and overlapping areas are blended. The accuracy of the proposed stitching scheme is evaluated on two datasets of 7 and 4 OCT volumes, respectively. By placing the specimens on a high-accuracy motorized translational stage, ground truth transformations are available. This results in a mean translational error between two adjacent volumes of 16.6 ± 0.8 μm (2.8 ± 0.13 voxels). To the author’s knowledge, this is the first reported stitching of multiple 3D OCT volumes by using dense voxel information in the registration process. The achieved results are sufficient for providing high accuracy OCT panoramic images. Combined with a recently available high-speed 4D OCT, our method enables interactive stitching of hand-guided acquisitions.

[Robotic Surgery - Who is The Boss?]

In the head and neck region, great potential is seen in robot-assisted surgery (RAS). Mainly in cancer surgery, the use of robotic systems seems to be of interest. Until today, two robotic systems (DaVinci® und FLEX®) have gained approval for clinical use in the head and neck region, and multiple other systems are currently in pre-clinical testing. Although, certain groups of patients may benefit from RAS, no unbiased randomized clinical studies are available. Until today, it was not possible to satisfactorily prove any advantage of RAS as compared to standard procedures. The limited clinical benefit and the additional financial burden seem to be the main reasons, why the comprehensive application of RAS has not been realized so far.This review article describes the large variety of clinical applications for RAS in the head and neck region. In addition, the financial and technical challenges, as well as ongoing developments of RAS are highlighted. Special focus is put on risks associated with RAS and current clinical studies. We believe, that RAS will find its way into clinical routine during the next years. Therefore, medical staff will have to increasingly face the technical, scientific and ethical features of RAS.

Surgical data science: The new knowledge domain

Healthcare in general, and surgery/interventional care in particular, is evolving through rapid advances in technology and increasing complexity of care, with the goal of maximizing the quality and value of care. Whereas innovations in diagnostic and therapeutic technologies have driven past improvements in the quality of surgical care, future transformation in care will be enabled by data. Conventional methodologies, such as registry studies, are limited in their scope for discovery and research, extent and complexity of data, breadth of analytical techniques, and translation or integration of research findings into patient care. We foresee the emergence of surgical/interventional data science (SDS) as a key element to addressing these limitations and creating a sustainable path toward evidence-based improvement of interventional healthcare pathways. SDS will create tools to measure, model, and quantify the pathways or processes within the context of patient health states or outcomes and use information gained to inform healthcare decisions, guidelines, best practices, policy, and training, thereby improving the safety and quality of healthcare and its value. Data are pervasive throughout the surgical care pathway; thus, SDS can impact various aspects of care, including prevention, diagnosis, intervention, or postoperative recovery. The existing literature already provides preliminary results, suggesting how a data science approach to surgical decision-making could more accurately predict severe complications using complex data from preoperative, intraoperative, and postoperative contexts, how it could support intraoperative decision-making using both existing knowledge and continuous data streams throughout the surgical care pathway, and how it could enable effective collaboration between human care providers and intelligent technologies. In addition, SDS is poised to play a central role in surgical education, for example, through objective assessments, automated virtual coaching, and robot-assisted active learning of surgical skill. However, the potential for transforming surgical care and training through SDS may only be realized through a cultural shift that not only institutionalizes technology to seamlessly capture data but also assimilates individuals with expertise in data science into clinical research teams. Furthermore, collaboration with industry partners from the inception of the discovery process promotes optimal design of data products as well as their efficient translation and commercialization. As surgery continues to evolve through advances in technology that enhance delivery of care, SDS represents a new knowledge domain to engineer surgical care of the future.