Passive markers for tracking surgical instruments in real-time 3-D ultrasound imaging

Extended reality for biomedicine

Extended reality (XR) refers to an umbrella of methods that allows users to be immersed in a three-dimensional (3D) or a 4D (spatial + temporal) virtual environment to different extents, including virtual reality (VR), augmented reality (AR), and mixed reality (MR). While VR allows a user to be fully immersed in a virtual environment, AR and MR overlay virtual objects over the real physical world. The immersion and interaction of XR provide unparalleled opportunities to extend our world beyond conventional lifestyles. While XR has extensive applications in fields such as entertainment and education, its numerous applications in biomedicine create transformative opportunities in both fundamental research and healthcare. This Primer outlines XR technology from instrumentation to software computation methods, delineating the biomedical applications that have been advanced by state-of-the-art techniques. We further describe the technical advances overcoming current limitations in XR and its applications, providing an entry point for professionals and trainees to thrive in this emerging field.

Recent Advances in Tracking Devices for Biomedical Ultrasound Imaging Applications

With the rapid advancement of tracking technologies, the applications of tracking systems in ultrasound imaging have expanded across a wide range of fields. In this review article, we discuss the basic tracking principles, system components, performance analyses, as well as the main sources of error for popular tracking technologies that are utilized in ultrasound imaging. In light of the growing demand for object tracking, this article explores both the potential and challenges associated with different tracking technologies applied to various ultrasound imaging applications, including freehand 3D ultrasound imaging, ultrasound image fusion, ultrasound-guided intervention and treatment. Recent development in tracking technology has led to increased accuracy and intuitiveness of ultrasound imaging and navigation with less reliance on operator skills, thereby benefiting the medical diagnosis and treatment. Although commercially available tracking systems are capable of achieving sub-millimeter resolution for positional tracking and sub-degree resolution for orientational tracking, such systems are subject to a number of disadvantages, including high costs and time-consuming calibration procedures. While some emerging tracking technologies are still in the research stage, their potentials have been demonstrated in terms of the compactness, light weight, and easy integration with existing standard or portable ultrasound machines.

Photoacoustic-based visual servoing of a needle tip

In intraoperative settings, the presence of acoustic clutter and reflection artifacts from metallic surgical tools often reduces the effectiveness of ultrasound imaging and complicates the localization of surgical tool tips. We propose an alternative approach for tool tracking and navigation in these challenging acoustic environments by augmenting ultrasound systems with a light source (to perform photoacoustic imaging) and a robot (to autonomously and robustly follow a surgical tool regardless of the tissue medium). The robotically controlled ultrasound probe continuously visualizes the location of the tool tip by segmenting and tracking photoacoustic signals generated from an optical fiber inside the tool. System validation in the presence of fat, muscle, brain, skull, and liver tissue with and without the presence of an additional clutter layer resulted in mean signal tracking errors <2 mm, mean probe centering errors <1 mm, and successful recovery from ultrasound perturbations, representing either patient motion or switching from photoacoustic images to ultrasound images to search for a target of interest. A detailed analysis of channel SNR in controlled experiments with and without significant acoustic clutter revealed that the detection of a needle tip is possible with photoacoustic imaging, particularly in cases where ultrasound imaging traditionally fails. Results show promise for guiding surgeries and procedures in acoustically challenging environments with this novel robotic and photoacoustic system combination.

An ergonomic handheld ultrasound probe providing contact forces and pose information

This paper presents a handheld ultrasound probe which is integrated with sensors to measure force and pose (position/orientation) information. Using an integrated probe like this, one can relate ultrasound images to spatial location and create 3D ultrasound maps. The handheld device can be used by sonographers and also easily be integrated with robot arms for automated sonography. The handheld device is ergonomically designed; rapid attachment and removal of the ultrasound transducer itself is possible using easy-to-operate clip mechanisms. A cable locking mechanism reduces the impact that gravitational and other external forces have (originating from data and power supply cables connected to the probe) on our measurements. Gravitational errors introduced by the housing of the probe are compensated for using knowledge of the housing geometry and the integrated pose sensor that provides us with accurate orientation information. In this paper, we describe the handheld probe with its integrated force/pose sensors and our approach to gravity compensation. We carried out a set of experiments to verify the feasibility of our approach to obtain accurate spatial information of the handheld probe.

Varying ultrasound power level to distinguish surgical instruments and tissue

We investigate a new framework of surgical instrument detection based on power-varying ultrasound images with simple and efficient pixel-wise intensity processing. Without using complicated feature extraction methods, we identified the instrument with an estimated optimal power level and by comparing pixel values of varying transducer power level images. The proposed framework exploits the physics of ultrasound imaging system by varying the transducer power level to effectively distinguish metallic surgical instruments from tissue. This power-varying image-guidance is motivated from our observations that ultrasound imaging at different power levels exhibit different contrast enhancement capabilities between tissue and instruments in ultrasound-guided robotic beating-heart surgery. Using lower transducer power levels (ranging from 40 to 75% of the rated lowest ultrasound power levels of the two tested ultrasound scanners) can effectively suppress the strong imaging artifacts from metallic instruments and thus, can be utilized together with the images from normal transducer power levels to enhance the separability between instrument and tissue, improving intraoperative instrument tracking accuracy from the acquired noisy ultrasound volumetric images. We performed experiments in phantoms and ex vivo hearts in water tank environments. The proposed multi-level power-varying ultrasound imaging approach can identify robotic instruments of high acoustic impedance from low-signal-to-noise-ratio ultrasound images by power adjustments.

Strategy for Monitoring Cardiac Interventions with an Intelligent Robotic Ultrasound Device

In recent years, 3D trans-oesophageal echocardiography (TOE) has become widely used for monitoring cardiac interventions. The control of the TOE probe during the procedure is a manual task which is tedious and harmful for the operator when exposed to radiation. To improve this technique, an add-on robotic system has been developed for holding and manipulating a commercial TOE probe. This paper focuses on the probe adjustment strategy in order to accurately monitor the moving intra-operative catheters. The positioning strategy is divided into an initialization step based on a pre-planning method, and a localized adjustment step based on the robotic differential kinematics. A series of experiments was performed to evaluate the initialization and the localized adjustment steps. The results indicate a mean error less than 10 mm from the phantom experiments for the initialization step, and a median error less than 1.5 mm from the computer-based simulation experiments for the localized adjustment step. Compared to the much bigger image volume, it is concluded that the proposed methods are feasible for this application. Future work will focus on evaluating the method in a more realistic TOE scanning scenario.

Medical Instrument Detection in 3-Dimensional Ultrasound Data Volumes

Ultrasound-guided medical interventions are broadly applied in diagnostics and therapy, e.g., regional anesthesia or ablation. A guided intervention using 2-D ultrasound is challenging due to the poor instrument visibility, limited field of view, and the multi-fold coordination of the medical instrument and ultrasound plane. Recent 3-D ultrasound transducers can improve the quality of the image-guided intervention if an automated detection of the needle is used. In this paper, we present a novel method for detecting medical instruments in 3-D ultrasound data that is solely based on image processing techniques and validated on various ex vivo and in vivo data sets. In the proposed procedure, the physician is placing the 3-D transducer at the desired position, and the image processing will automatically detect the best instrument view, so that the physician can entirely focus on the intervention. Our method is based on the classification of instrument voxels using volumetric structure directions and robust approximation of the primary tool axis. A novel normalization method is proposed for the shape and intensity consistency of instruments to improve the detection. Moreover, a novel 3-D Gabor wavelet transformation is introduced and optimally designed for revealing the instrument voxels in the volume, while remaining generic to several medical instruments and transducer types. Experiments on diverse data sets, including in vivo data from patients, show that for a given transducer and an instrument type, high detection accuracies are achieved with position errors smaller than the instrument diameter in the 0.5-1.5-mm range on average.

A new device for fiducial registration of image-guided navigation system for liver RFA

PURPOSE

Radiofrequency ablation for liver tumors (liver RFA) is widely performed under ultrasound guidance. However, discriminating between the tumor and the needle is often difficult because of cavitation caused by RFA-induced coagulation. An unclear ultrasound image can lead to complications and tumor residue. Therefore, image-guided navigation systems based on fiducial registration have been developed. Fiducial points are usually set on a patient's skin. But the use of internal fiducial points can improve the accuracy of navigation. In this study, a new device is introduced to use internal fiducial points using 2D US.

METHODS

3D Slicer as the navigation software, Polaris Vicra as the position sensor, and two target tumors in a 3D abdominal phantom as puncture targets were used. Also, a new device that makes it possible to obtain tracking coordinates in the body was invented. First, two-dimensional reslice images from the CT images using 3D Slicer were built. A virtual needle was displayed on the two-dimensional reslice image, reflecting the movement of the actual needle after fiducial registration. A phantom experiment using three sets of fiducial point configurations: one conventional case using only surface points, and two cases in which the center of the target tumor was selected as a fiducial point was performed. For each configuration, one surgeon punctured each target tumor ten times under guidance from the 3D Slicer display. Finally, a statistical analysis examining the puncture error was performed.

RESULTS

The puncture error for each target tumor decreased significantly when the center of the target tumor was included as one of the fiducial points, compared with when only surface points were used.

CONCLUSION

This study introduces a new device to use internal fiducial points and suggests that the accuracy of image-guided navigation systems for liver RFA can be improved by using the new device.

Model predictive control of a robotically actuated delivery sheath for beating heart compensation

Minimally invasive surgery (MIS) during cardiovascular interventions reduces trauma and enables the treatment of high-risk patients who were initially denied surgery. However, restricted access, reduced visibility and control of the instrument at the treatment locations limits the performance and capabilities of such interventions during MIS. Therefore, the demand for technology such as steerable sheaths or catheters that assist the clinician during the procedure is increasing. In this study, we present and evaluate a robotically actuated delivery sheath (RADS) capable of autonomously and accurately compensating for beating heart motions by using a model-predictive control (MPC) strategy. We develop kinematic models and present online ultrasound segmentation of the RADS that are integrated with the MPC strategy. As a case study, we use pre-operative ultrasound images from a patient to extract motion profiles of the aortic heart valve (AHV). This allows the MPC strategy to anticipate for AHV motions. Further, mechanical hysteresis in the steering mechanism is compensated for in order to improve tip positioning accuracy. The novel integrated system is capable of controlling the articulating tip of the RADS to assist the clinician during cardiovascular surgery. Experiments demonstrate that the RADS follows the AHV motion with a mean positioning error of 1.68 mm. The presented modelling, imaging and control framework could be adapted and applied to a range of continuum-style robots and catheters for various cardiovascular interventions.

Spatial calibration of a 2D/3D ultrasound using a tracked needle

Purpose

Spatial calibration between a 2D/3D ultrasound and a pose tracking system requires a complex and time-consuming procedure. Simplifying this procedure without compromising the calibration accuracy is still a challenging problem.

Method

We propose a new calibration method for both 2D and 3D ultrasound probes that involves scanning an arbitrary region of a tracked needle in different poses. This approach is easier to perform than most alternative methods that require a precise alignment between US scans and a calibration phantom.

Results

Our calibration method provides an average accuracy of 2.49 mm for a 2D US probe with 107 mm scanning depth, and an average accuracy of 2.39 mm for a 3D US with 107 mm scanning depth.

Conclusion

Our method proposes a unified calibration framework for 2D and 3D probes using the same phantom object, work-flow, and algorithm. Our method significantly improves the accuracy of needle-based methods for 2D US probes as well as extends its use for 3D US probes.

Treatment planning and image guidance for radiofrequency ablation of large tumors

This article addresses the two key challenges in computer-assisted percutaneous tumor ablation: planning multiple overlapping ablations for large tumors while avoiding critical structures, and executing the prescribed plan. Toward semiautomatic treatment planning for image-guided surgical interventions, we develop a systematic approach to the needle-based ablation placement task, ranging from preoperative planning algorithms to an intraoperative execution platform. The planning system incorporates clinical constraints on ablations and trajectories using a multiple objective optimization formulation, which consists of optimal path selection and ablation coverage optimization based on integer programming. The system implementation is presented and validated in both phantom and animal studies. The presented system can potentially be further extended for other ablation techniques such as cryotherapy.

ROBOTICS IN NATURAL ORIFICE TRANSLUMINAL ENDOSCOPIC SURGERY

Natural orifice translumenal endoscopic surgery (NOTES) is the latest surgery paradigm in which the abdominal cavity is accessed via the body's natural orifice, e.g., vagina, mouth, etc. Compared with traditional laparoscopic surgery, NOTES completely eliminates the skin incision and therefore benefits the patients in several aspects such as less post-operative pain, shorter recovery period, fewer complications, etc. Due to the unique characteristics of NOTES, instruments for traditional laparoscopic surgery are not suitable for NOTES and hence novel hardware design is necessary for facilitating system development. This paper gives an overview of the state of the arts in the development of surgical instruments for NOTES, particularly with a focus on the promising robotic endoscopes.

Ultrasound image-based endoscope localization for minimally invasive fetoscopic surgery

The purpose of this work is to introduce an ultrasound image-based intraoperative scheme for rigid endoscope localization during minimally invasive fetoscopic surgery. Positional information of surgical instruments with respect to anatomical features is important for the development of computer-aided surgery applications. While most surgical navigation systems use optical tracking systems with satisfactory accuracy, there are several operation limitations in such systems. We propose an elegant framework for intraoperative instrument localization that does not require any external tracking system but uses an ultrasound imaging system and a computation scheme based on constrained kinematics of minimally invasive fetoscopic surgery. Our proposed algorithm simultaneously estimates endoscope and port positions in an online sequential fashion with standard deviation of 1.28 mm for port estimation. Robustness of the port estimation algorithm against external disturbance was demonstrated by intentionally introducing artificial errors to measurement data. The estimation converges within eight iterations under disturbance magnitude of 30 mm.

Robotics and imaging in congenital heart surgery

The initial success seen in adult cardiac surgery with the application of available robotic systems has not been realized as broadly in pediatric cardiac surgery. The main obstacles include extended set-up time and complexity of the procedures, as well as the large size of the instruments with respect to the size of the child. Moreover, while the main advantage of robotic systems is the ability to minimize incision size, for intracardiac repairs, cardiopulmonary bypass is still required. Catheter-based interventions, on the other hand, have expanded rapidly in both application as well as the complexity of procedures and lesions being treated. However, despite the development of sophisticated devices, robotic systems to aid catheter procedures have not been commonly applied in children. In this article, we describe new catheter-like robotic delivery platforms, which facilitate safe navigation and enable complex repairs, such as tissue approximation and fixation, and tissue removal, inside the beating heart. Additional features including the tracking of rapidly moving tissue targets and novel imaging approaches are described, along with a discussion of future prospects for steerable robotic systems.