Needle Segmentation in Volumetric Optical Coherence Tomography Images for Ophthalmic Microsurgery

Robotising vitreoretinal surgeries

The use of robotic surgery in ophthalmology has been shown to offer many potential advantages to current surgical techniques. Vitreoretinal surgery requires complex manoeuvres and high precision, and this is an area that exceeds manual human dexterity in certain surgical situations. With the advent of advanced therapeutics such as subretinal gene therapy, precise delivery and minimising trauma is imperative to optimize outcomes. There are multiple robotic systems in place for ophthalmology in pre-clinical and clinical use, and the Preceyes Robotic Surgical System (Preceyes BV) has also gained the CE mark and is commercially available for use. Recent in-vivo and in-human surgeries have been performed successfully with robotics systems. This includes membrane peeling, subretinal injections of therapeutics, and retinal vein cannulation. There is huge potential to integrate robotic surgery into mainstream clinical practice. In this review, we summarize the existing systems, and clinical implementation so far, and highlight the future clinical applications for robotic surgery in vitreo-retina.

Autonomous Needle Navigation in Subretinal Injections via iOCT

Subretinal injection is an effective method for direct delivery of therapeutic agents to treat prevalent subretinal diseases. Among the challenges for surgeons are physiological hand tremor, difficulty resolving single-micron scale depth perception, and lack of tactile feedback. The recent introduction of intraoperative Optical Coherence Tomography (iOCT) enables precise depth information during subretinal surgery. However, even when relying on iOCT, achieving the required micron-scale precision remains a significant surgical challenge. This work presents a robot-assisted workflow for high-precision autonomous needle navigation for subretinal injection. The workflow includes online registration between robot and iOCT coordinates; tool-tip localization in iOCT coordinates using a Convolutional Neural Network (CNN); and tool-tip planning and tracking system using real-time Model Predictive Control (MPC). The proposed workflow is validated using a silicone eye phantom and ex vivo porcine eyes. The experimental results demonstrate that the mean error to reach the user-defined target and the mean procedure duration are within an acceptable precision range. The proposed workflow achieves a 100% success rate for subretinal injection, while maintaining scleral forces at the scleral insertion point below 15mN throughout the navigation procedures.

Robotic Navigation Autonomy for Subretinal Injection via Intelligent Real-Time Virtual iOCT Volume Slicing

In the last decade, various robotic platforms have been introduced that could support delicate retinal surgeries. Concurrently, to provide semantic understanding of the surgical area, recent advances have enabled microscope-integrated intraoperative Optical Coherent Tomography (iOCT) with high-resolution 3D imaging at near video rate. The combination of robotics and semantic understanding enables task autonomy in robotic retinal surgery, such as for subretinal injection. This procedure requires precise needle insertion for best treatment outcomes. However, merging robotic systems with iOCT introduces new challenges. These include, but are not limited to high demands on data processing rates and dynamic registration of these systems during the procedure. In this work, we propose a framework for autonomous robotic navigation for subretinal injection, based on intelligent real-time processing of iOCT volumes. Our method consists of an instrument pose estimation method, an online registration between the robotic and the iOCT system, and trajectory planning tailored for navigation to an injection target. We also introduce intelligent virtual B-scans, a volume slicing approach for rapid instrument pose estimation, which is enabled by Convolutional Neural Networks (CNNs). Our experiments on ex-vivo porcine eyes demonstrate the precision and repeatability of the method. Finally, we discuss identified challenges in this work and suggest potential solutions to further the development of such systems.

Theoretical error analysis of spotlight-based instrument localization for retinal surgery

Retinal surgery is widely considered to be a complicated and challenging task even for specialists. Image-guided robot-assisted intervention is among the novel and promising solutions that may enhance human capabilities therein. In this paper, we demonstrate the possibility of using spotlights for 5D guidance of a microsurgical instrument. The theoretical basis of the localization for the instrument based on the projection of a single spotlight is analyzed to deduce the position and orientation of the spotlight source. The usage of multiple spotlights is also proposed to check the possibility of further improvements for the performance boundaries. The proposed method is verified within a high-fidelity simulation environment using the 3D creation suite Blender. Experimental results show that the average positioning error is 0.029 mm using a single spotlight and 0.025 mm with three spotlights, respectively, while the rotational errors are 0.124 and 0.101, which shows the application to be promising in instrument localization for retinal surgery.

Simultaneous Online Registration-Independent Stiffness Identification and Tip Localization of Surgical Instruments in Robot-assisted Eye Surgery

Notable challenges during retinal surgery lend themselves to robotic assistance which has proven beneficial in providing a safe steady-hand manipulation. Efficient assistance from the robots heavily relies on accurate sensing of surgery states (e.g. instrument tip localization and tool-to-tissue interaction forces). Many of the existing tool tip localization methods require preoperative frame registrations or instrument calibrations. In this study using an iterative approach and by combining vision and force-based methods, we develop calibration- and registration-independent (RI) algorithms to provide online estimates of instrument stiffness (least squares and adaptive). The estimations are then combined with a state-space model based on the forward kinematics (FWK) of the Steady-Hand Eye Robot (SHER) and Fiber Brag Grating (FBG) sensor measurements. This is accomplished using a Kalman Filtering (KF) approach to improve the deflected instrument tip position estimations during robot-assisted eye surgery. The conducted experiments demonstrate that when the online RI stiffness estimations are used, the instrument tip localization results surpass those obtained from pre-operative offline calibrations for stiffness.

Surgical scene generation and adversarial networks for physics-based iOCT synthesis

The development and integration of intraoperative optical coherence tomography (iOCT) into modern operating rooms has motivated novel procedures directed at improving the outcome of ophthalmic surgeries. Although computer-assisted algorithms could further advance such interventions, the limited availability and accessibility of iOCT systems constrains the generation of dedicated data sets. This paper introduces a novel framework combining a virtual setup and deep learning algorithms to generate synthetic iOCT data in a simulated environment. The virtual setup reproduces the geometry of retinal layers extracted from real data and allows the integration of virtual microsurgical instrument models. Our scene rendering approach extracts information from the environment and considers iOCT typical imaging artifacts to generate cross-sectional label maps, which in turn are used to synthesize iOCT B-scans via a generative adversarial network. In our experiments we investigate the similarity between real and synthetic images, show the relevance of using the generated data for image-guided interventions and demonstrate the potential of 3D iOCT data synthesis.

Spotlight-based 3D Instrument Guidance for Retinal Surgery

Retinal surgery is a complex activity that can be challenging for a surgeon to perform effectively and safely. Image guided robot-assisted surgery is one of the promising solutions that bring significant surgical enhancement in treatment outcome and reduce the physical limitations of human surgeons. In this paper, we demonstrate a novel method for 3D guidance of the instrument based on the projection of spotlight in the single microscope images. The spotlight projection mechanism is firstly analyzed and modeled with a projection on both a plane and a sphere surface. To test the feasibility of the proposed method, a light fiber is integrated into the instrument which is driven by the Steady-Hand Eye Robot (SHER). The spot of light is segmented and tracked on a phantom retina using the proposed algorithm. The static calibration and dynamic test results both show that the proposed method can easily archive 0.5 mm of tip-to-surface distance which is within the clinically acceptable accuracy for intraocular visual guidance.

Real-time tool to layer distance estimation for robotic subretinal injection using intraoperative 4D OCT

The emergence of robotics could enable ophthalmic microsurgical procedures that were previously not feasible due to the precision limits of manual delivery, for example, targeted subretinal injection. Determining the distance between the needle tip, the internal limiting membrane (ILM), and the retinal pigment epithelium (RPE) both precisely and reproducibly is required for safe and successful robotic retinal interventions. Recent advances in intraoperative optical coherence tomography (iOCT) have opened the path for 4D image-guided surgery by providing near video-rate imaging with micron-level resolution to visualize retinal structures, surgical instruments, and tool-tissue interactions. In this work, we present a novel pipeline to precisely estimate the distance between the injection needle and the surface boundaries of two retinal layers, the ILM and the RPE, from iOCT volumes. To achieve high computational efficiency, we reduce the analysis to the relevant area around the needle tip. We employ a convolutional neural network (CNN) to segment the tool surface, as well as the retinal layer boundaries from selected iOCT B-scans within this tip area. This results in the generation and processing of 3D surface point clouds for the tool, ILM and RPE from the B-scan segmentation maps, which in turn allows the estimation of the minimum distance between the resulting tool and layer point clouds. The proposed method is evaluated on iOCT volumes from ex-vivo porcine eyes and achieves an average error of 9.24 µm and 8.61 µm measuring the distance from the needle tip to the ILM and the RPE, respectively. The results demonstrate that this approach is robust to the high levels of noise present in iOCT B-scans and is suitable for the interventional use case by providing distance feedback at an average update rate of 15.66 Hz.

Advanced robotic surgical systems in ophthalmology

In this paper, an overview of advanced robotic surgical systems in ophthalmology is provided. The systems are introduced as representative examples of the degree of human vs. robotic control during surgical procedures. The details are presented on each system and the latest advancements of each are described. Future potential applications for surgical robotics in ophthalmology are discussed in detail, with representative examples provided alongside recent progress.

CATARACTS: Challenge on automatic tool annotation for cataRACT surgery

Surgical tool detection is attracting increasing attention from the medical image analysis community. The goal generally is not to precisely locate tools in images, but rather to indicate which tools are being used by the surgeon at each instant. The main motivation for annotating tool usage is to design efficient solutions for surgical workflow analysis, with potential applications in report generation, surgical training and even real-time decision support. Most existing tool annotation algorithms focus on laparoscopic surgeries. However, with 19 million interventions per year, the most common surgical procedure in the world is cataract surgery. The CATARACTS challenge was organized in 2017 to evaluate tool annotation algorithms in the specific context of cataract surgery. It relies on more than nine hours of videos, from 50 cataract surgeries, in which the presence of 21 surgical tools was manually annotated by two experts. With 14 participating teams, this challenge can be considered a success. As might be expected, the submitted solutions are based on deep learning. This paper thoroughly evaluates these solutions: in particular, the quality of their annotations are compared to that of human interpretations. Next, lessons learnt from the differential analysis of these solutions are discussed. We expect that they will guide the design of efficient surgery monitoring tools in the near future.

Fast 5DOF needle tracking in iOCT

PURPOSE

Intraoperative optical coherence tomography (iOCT) is an increasingly available imaging technique for ophthalmic microsurgery that provides high-resolution cross-sectional information of the surgical scene. We propose to build on its desirable qualities and present a method for tracking the orientation and location of a surgical needle. Thereby, we enable the direct analysis of instrument-tissue interaction directly in OCT space without complex multimodal calibration that would be required with traditional instrument tracking methods.

METHOD

The intersection of the needle with the iOCT scan is detected by a peculiar multistep ellipse fitting that takes advantage of the directionality of the modality. The geometric modeling allows us to use the ellipse parameters and provide them into a latency-aware estimator to infer the 5DOF pose during needle movement.

RESULTS

Experiments on phantom data and ex vivo porcine eyes indicate that the algorithm retains angular precision especially during lateral needle movement and provides a more robust and consistent estimation than baseline methods.

CONCLUSION

Using solely cross-sectional iOCT information, we are able to successfully and robustly estimate a 5DOF pose of the instrument in less than 5.4 ms on a CPU.

Towards Robot-Assisted Retinal Vein Cannulation: A Motorized Force-Sensing Microneedle Integrated with a Handheld Micromanipulator †

Retinal vein cannulation is a technically demanding surgical procedure where therapeutic agents are injected into the retinal veins to treat occlusions. The clinical feasibility of this approach has been largely limited by the technical challenges associated with performing the procedure. Among the challenges to successful vein cannulation are identifying the moment of venous puncture, achieving cannulation of the micro-vessel, and maintaining cannulation throughout drug delivery. Recent advances in medical robotics and sensing of tool-tissue interaction forces have the potential to address each of these challenges as well as to prevent tissue trauma, minimize complications, diminish surgeon effort, and ultimately promote successful retinal vein cannulation. In this paper, we develop an assistive system combining a handheld micromanipulator, called "Micron", with a force-sensing microneedle. Using this system, we examine two distinct methods of precisely detecting the instant of venous puncture. This is based on measured tool-tissue interaction forces and also the tracked position of the needle tip. In addition to the existing tremor canceling function of Micron, a new control method is implemented to actively compensate unintended movements of the operator, and to keep the cannulation device securely inside the vein following cannulation. To demonstrate the capabilities and performance of our uniquely upgraded system, we present a multi-user artificial phantom study with subjects from three different surgical skill levels. Results show that our puncture detection algorithm, when combined with the active positive holding feature enables sustained cannulation which is most evident in smaller veins. Notable is that the active holding function significantly attenuates tool motion in the vein, thereby reduces the trauma during cannulation.