3-D ultrasound-guided robotic needle steering in biological tissue

Color VISION for improved ultrasound visualization of brachytherapy needles

BACKGROUND

High dose rate brachytherapy is commonly used in the treatment of prostate cancer. Treatment planning is often performed under transrectal ultrasound (US) guidance, but brachytherapy needles can be challenging to digitize due to the presence of poor US conspicuity and imaging artifacts. The plan accuracy and quality, however, are dependent on the proper visualization of the needles with millimeter accuracy.

PURPOSE

This work describes a technique for generating a color overlay of needle locations atop the grayscale US image. Prototype devices were developed to produce vibrations in the brachytherapy needles that generate a high contrast color Doppler (CD) signal that highlights the needle locations with superior contrast and reduced artifacts. Denoted by the acronym color VISION (Vibrationally Induced Shimmering for Identifying an Object's Nature), the technology has the potential to improve applicator conspicuity and facilitate automated applicator digitization.

METHODS

Three prototype vibrational devices with frequencies between 200-450 Hz were designed in-house and evaluated with needle implants in a phantom and cadaveric male pelvis using: (1) an actuator attached to the front of a prostate needle template; (2) an actuator attached to the top of the needle template; and (3) a hand-held actuator with a stylet, inserted directly into a needle's inner lumen. Acquired images were postprocessed in MATLAB to evaluate the potential for automated digitization.

RESULTS

All prototype devices produced localized shimmering in implanted brachytherapy needles in both the axial and sagittal planes. The template mounted actuators provided better vibrational coupling and ease of operation than the stylet prototype. The Michelson contrast, or visibility, of the shimmering CD signal was 100% compared with ≤40% for B-mode imaging of a single needle. Proof-of-principle for automated applicator digitization using only the CD signal was demonstrated.

CONCLUSIONS

The color VISION prototype devices successfully coupled mechanical vibrations into brachytherapy needles to generate US CD shimmering and accurately highlight brachytherapy needle locations. The high contrast and natively registered signal are promising for future work to automate the needle digitization and provide a real-time visual overlay of the applicator on the B-mode US image.

Hybrid Deep Learning and Model-Based Needle Shape Prediction

Needle insertion using flexible bevel tip needles are a common minimally-invasive surgical technique for prostate cancer interventions. Flexible, asymmetric bevel tip needles enable physicians for complex needle steering techniques to avoid sensitive anatomical structures during needle insertion. For accurate placement of the needle, predicting the trajectory of these needles intra-operatively would greatly reduce the need for frequently needle reinsertions thus improving patient comfort and positive outcomes. However, predicting the trajectory of the needle during insertion is a complex task that has yet to be solved due to random needle-tissue interactions. In this paper, we present and validate for the first time a hybrid deep learning and model-based approach to handle the intra-operative needle shape prediction problem through, leveraging a validated Lie-group theoretic model for needle shape representation. Furthermore, we present a novel self-supervised learning and method in conjunction with the Lie-group shape model for training these networks in the absence of data, enabling further refinement of these networks with transfer learning. Needle shape prediction was performed in single-layer and double-layer homogeneous phantom tissue for C- and S-shape needle insertions. Our method demonstrates an average root-mean-square prediction error of 1.03 mm over a dataset containing approximately 3,000 prediction samples with maximum prediction steps of 110 mm.

A Survey of Needle Steering Approaches in Minimally Invasive Surgery

In virtue of a curved insertion path inside tissues, needle steering techniques have revealed the potential with the assistance of medical robots and images. The superiority of this technique has been preliminarily verified with several maneuvers: target realignment, obstacle circumvention, and multi-target access. However, the momentum of needle steering approaches in the past decade leads to an open question-"How to choose an applicable needle steering approach for a specific clinical application?" This survey discusses this question in terms of design choices and clinical considerations, respectively. In view of design choices, this survey proposes a hierarchical taxonomy of current needle steering approaches. Needle steering approaches of different manipulations and designs are classified to systematically review the design choices and their influences on clinical treatments. In view of clinical consideration, this survey discusses the steerability and acceptability of the current needle steering approaches. On this basis, the pros and cons of the current needle steering approaches are weighed and their suitable applications are summarized. At last, this survey concluded with an outlook of the needle steering techniques, including the potential clinical applications and future developments in mechanical design.

Optical Fiber-Based Needle Shape Sensing in Real Tissue: Single Core vs. Multicore Approaches

Flexible needle insertion procedures are common in minimally-invasive surgeries for diagnosing and treating prostate cancer. Bevel-tip needles provide physicians the capability to steer the needle during long insertions to avoid vital anatomical structures in the patient and reduce post-operative patient discomfort. To provide needle placement feedback to the physician, sensors are embedded into needles for determining the real-time 3D shape of the needle during operation without needing to visualize the needle intra-operatively. Through expansive research in fiber optics, a plethora of bio-compatible, MRI-compatible, optical shape-sensors have been developed to provide real-time shape feedback, such as single-core and multicore fiber Bragg gratings. In this paper, we directly compare single-core fiber-based and multicore fiber-based needle shape-sensing through similarly constructed, four-active area sensorized bevel-tip needles inserted into phantom and ex-vivo tissue on the same experimental platform. In this work, we found that for shape-sensing in phantom tissue, the two needles performed identically with a p-value of 0.164 > 0.05, but in ex-vivo real tissue, the single-core fiber sensorized needle significantly outperformed the multicore fiber configuration with a p-value of 0.0005 < 0.05. This paper also presents the experimental platform and method for directly comparing these optical shape sensors for the needle shape-sensing task, as well as provides direction, insight and required considerations for future work in constructively optimizing sensorized needles.

Robotic ultrasound imaging: State-of-the-art and future perspectives

Ultrasound (US) is one of the most widely used modalities for clinical intervention and diagnosis due to the merits of providing non-invasive, radiation-free, and real-time images. However, free-hand US examinations are highly operator-dependent. Robotic US System (RUSS) aims at overcoming this shortcoming by offering reproducibility, while also aiming at improving dexterity, and intelligent anatomy and disease-aware imaging. In addition to enhancing diagnostic outcomes, RUSS also holds the potential to provide medical interventions for populations suffering from the shortage of experienced sonographers. In this paper, we categorize RUSS as teleoperated or autonomous. Regarding teleoperated RUSS, we summarize their technical developments, and clinical evaluations, respectively. This survey then focuses on the review of recent work on autonomous robotic US imaging. We demonstrate that machine learning and artificial intelligence present the key techniques, which enable intelligent patient and process-specific, motion and deformation-aware robotic image acquisition. We also show that the research on artificial intelligence for autonomous RUSS has directed the research community toward understanding and modeling expert sonographers' semantic reasoning and action. Here, we call this process, the recovery of the "language of sonography". This side result of research on autonomous robotic US acquisitions could be considered as valuable and essential as the progress made in the robotic US examination itself. This article will provide both engineers and clinicians with a comprehensive understanding of RUSS by surveying underlying techniques. Additionally, we present the challenges that the scientific community needs to face in the coming years in order to achieve its ultimate goal of developing intelligent robotic sonographer colleagues. These colleagues are expected to be capable of collaborating with human sonographers in dynamic environments to enhance both diagnostic and intraoperative imaging.

Optical Fiber-Based Needle Shape Sensing in Real Tissue: Single Core vs. Multicore Approaches

Flexible needle insertion procedures are common for minimally-invasive surgeries for diagnosing and treating prostate cancer. Bevel-tip needles provide physicians the capability to steer the needle during long insertions to avoid vital anatomical structures in the patient and reduce post-operative patient discomfort. To provide needle placement feedback to the physician, sensors are embedded into needles for determining the real-time 3D shape of the needle during operation without needing to visualize the needle intra-operatively. Through expansive research in fiber optics, a plethora of bio-compatible, MRI-compatible, optical shape-sensors have been developed to provide real-time shape feedback, such as single-core and multicore fiber Bragg gratings. In this paper, we directly compare single-core fiber-based and multicore fiber-based needle shape-sensing through identically constructed, four-active area sensorized bevel-tip needles inserted into phantom and ex-vivo tissue on the same experimental platform. In this work, we found that for shape-sensing in phantom tissue, the two needles performed identically with a p -value of 0.164 > 0.05, but in ex-vivo real tissue, the single-core fiber sensorized needle significantly outperformed the multicore fiber configuration with a p -value of 0.0005 < 0.05. This paper also presents the experimental platform and method for directly comparing these optical shape sensors for the needle shape-sensing task, as well as provides direction, insight and required considerations for future work in constructively optimizing sensorized needles.

Robot-Guided Ultrasonography in Surgical Interventions

INTRODUCTION

The introduction of robotic-guided procedures in surgical techniques has brought an increase in the accuracy and control of resections. Surgery has evolved as a technique since the development of laparoscopy, which has added to the visualisation of the peritoneal cavity from a different perspective. Multi-armed robot associated with real-time intraoperative imaging devices brings important manoeuvrability and dexterity improvements in certain surgical fields.

MATERIALS AND METHODS

The present study is designed to synthesise the development of imaging techniques with a focus on ultrasonography in robotic surgery in the last ten years regarding abdominal surgical interventions.

RESULTS

All studies involved abdominal surgery. Out of the seven studies, two were performed in clinical trials. The other five studies were performed on organs or simulators and attempted to develop a hybrid surgical technique using ultrasonography and robotic surgery. Most studies aim to surgically identify both blood vessels and nerve structures through this combined technique (surgery and imaging).

CONCLUSIONS

Ultrasonography is often used in minimally invasive surgical techniques. This adds to the visualisation of blood vessels, the correct identification of tumour margins, and the location of surgical instruments in the tissue. The development of ultrasound technology from 2D to 3D and 4D has brought improvements in minimally invasive and robotic surgical techniques, and it should be further studied to bring surgery to a higher level.

Optical Fiber -Based Needle Shape Sensing: Three-channel Single Core vs. Multicore Approaches

Bevel-tip needles are commonly utilized in percutaneous medical interventions where a curved insertion trajectory is required. To avoid deviation from the intended trajectory, needle shape sensing and tip localization is crucial in providing the operator with feedback. There is an abundance of previous work that investigate the medical application of fiber Bragg grating (FBG) sensors, but most works select only one specific type of fiber among the many available sensor options to integrate into their hardware designs. In this work, we compare two different types of FBG sensors under identical conditions and application, namely, acting as the sensor for needle insertion shape reconstruction. We built a three-channel single core needle and a seven-channel multicore fiber (MCF) needle and discuss the pros and cons of both constructions for shape sensing experiments into constant curvature jigs. The overall needle tip error is 1.23 mm for the single core needle and 2.08 mm for the multicore needle.

Flexible Needle Steering with Tethered and Untethered Actuation: Current States, Targeting Errors, Challenges and Opportunities

Accurate needle targeting is critical for many clinical procedures, such as transcutaneous biopsy or radiofrequency ablation of tumors. However, targeting errors may arise, limiting the widespread adoption of these procedures. Advances in flexible needle (FN) steering are emerging to mitigate these errors. This review summarizes the state-of-the-art developments of FNs and addresses possible targeting errors that can be overcome with steering actuation techniques. FN steering techniques can be classified as passive and active. Passive steering directly results from the needle-tissue interaction forces, whereas active steering requires additional forces to be applied at the needle tip, which enhances needle steerability. Therefore, the corresponding targeting errors of most passive FNs and active FNs are between 1 and 2 mm, and less than 1 mm, respectively. However, the diameters of active FNs range from 1.42 to 12 mm, which is larger than the passive steering needle varying from 0.5 to 1.4 mm. Therefore, the development of active FNs is an area of active research. These active FNs can be steered using tethered internal direct actuation or untethered external actuation. Examples of tethered internal direct actuation include tendon-driven, longitudinal segment transmission and concentric tube transmission. Tendon-driven FNs have various structures, and longitudinal segment transmission needles could be adapted to reduce tissue damage. Additionally, concentric tube needles have immediate advantages and clinical applications in natural orifice surgery. Magnetic actuation enables active FN steering with untethered external actuation and facilitates miniaturization. The challenges faced in the fabrication, sensing, and actuation methods of FN are analyzed. Finally, bio-inspired FNs may offer solutions to address the challenges faced in FN active steering mechanisms.

A power Doppler ultrasound method for improving intraoperative tip localization for visually obstructed needles in interstitial prostate brachytherapy

PURPOSE

High-dose-rate (HDR) interstitial brachytherapy (BT) is a common treatment technique for localized intermediate to high-risk prostate cancer. Transrectal ultrasound (US) imaging is typically used for guiding needle insertion, including localization of the needle tip which is critical for treatment planning. However, image artifacts can limit needle tip visibility in standard brightness (B)-mode US, potentially leading to dose delivery that deviates from the planned dose. To improve intraoperative tip visualization in visually obstructed needles, we propose a power Doppler (PD) US method which utilizes a novel wireless mechanical oscillator, validated in phantom experiments and clinical HDR-BT cases as part of a feasibility clinical trial.

METHODS

Our wireless oscillator contains a DC motor housed in a 3D printed case and is powered by rechargeable battery allowing the device to be operated by one person with no additional equipment required in the operating room. The oscillator end-piece features a cylindrical shape designed for BT applications to fit on top of the commonly used cylindrical needle mandrins. Phantom validation was completed using tissue-equivalent agar phantoms with the clinical US system and both plastic and metal needles. Our PD method was tested using a needle implant pattern matching a standard HDR-BT procedure as well as an implant pattern designed to maximize needle shadowing artifacts. Needle tip localization accuracy was assessed using the clinical method based on ideal reference needles as well as a comparison to computed tomography (CT) as a gold standard. Clinical validation was completed in five patients who underwent standard HDR-BT as part of a feasibility clinical trial. Needle tips positions were identified using B-mode US and PD US with perturbation from our wireless oscillator.

RESULTS

Absolute mean ± standard deviation tip error for B-mode alone, PD alone, and B-mode combined with PD was respectively: 0.3 ± 0.3 mm, 0.6 ± 0.5 mm, and 0.4 ± 0.2 mm for the mock HDR-BT needle implant; 0.8 ± 1.7 mm, 0.4 ± 0.6 mm, and 0.3 ± 0.5 mm for the explicit shadowing implant with plastic needles; and 0.5 ± 0.2 mm, 0.5 ± 0.3 mm, and 0.6 ± 0.2 mm for the explicit shadowing implant with metal needles. The total mean absolute tip error for all five patients in the feasibility clinical trial was 0.9 ± 0.7 mm using B-mode US alone and 0.8 ± 0.5 mm when including PD US, with increased benefit observed for needles classified as visually obstructed.

CONCLUSIONS

Our proposed PD needle tip localization method is easy to implement and requires no modifications or additions to the standard clinical equipment or workflow. We have demonstrated decreased tip localization error and variation for visually obstructed needles in both phantom and clinical cases, including providing the ability to visualize needles previously not visible using B-mode US alone. This method has the potential to improve needle visualization in challenging cases without burdening the clinical workflow, potentially improving treatment accuracy in HDR-BT and more broadly in any minimally invasive needle-based procedure.

Lie-Group Theoretic Approach to Shape-Sensing Using FBG-Sensorized Needles Including Double-Layer Tissue and S-Shape Insertions

Flexible bevel-tipped needles are often used for needle insertion in minimally-invasive surgical techniques due to their ability to be steered in cluttered environments. Shapesensing enables physicians to determine the location of needles intra-operatively without requiring radiation of the patient, enabling accurate needle placement. In this paper, we validate a theoretical method for flexible needle shape-sensing that allows for complex curvatures, extending upon a previous sensor-based model. This model combines curvature measurements from fiber Bragg grating (FBG) sensors and the mechanics of an inextensible elastic rod to determine and predict the 3D needle shape during insertion. We evaluate the model's shape sensing capabilities in C- and S-shape insertions in single-layer isotropic tissue, and C-shape insertions in two-layer isotropic tissue. Experiments on a four-active area, FBG-sensorized needle were performed in varying tissue stiffnesses and insertion scenarios under stereo vision to provide the 3D ground truth needle shape. The results validate a viable 3D needle shape-sensing model accounting for complex curvatures in flexible needles with mean needle shape sensing root-mean-square errors of 0.160 ± 0.055 mm over 650 needle insertions.

Concentric-ring arrays for forward-viewing ultrasound imaging

Purpose

Current ultrasound (US)-image-guided needle insertions often require an expertized technique for clinicians because the performance of tasks in a three-dimensional space using two-dimensional images requires operators to cognitively maintain the spatial relationships between the US probe, the needle, and the lesion. This work presents forward-viewing US imaging with a ring array configuration to enable needle interventions without requiring the registration between tools and targets.

Approach

The center-open ring array configuration allows the needle to be inserted from the center of the visualized US image, providing simple and intuitive guidance. To establish the feasibility of the ring array configuration, the design parameters causing the image quality, including the radius of the center hole and the number of ring layers and transducer elements, were investigated.

Results

Experimental results showed successful visualization, even with a hole in the transducer elements, and the target visibility was improved by increasing the number of ring layers and the number of transducer elements in each ring layer. Reducing the hole radius improved the region's image quality at a shallow depth.

Conclusions

Forward-viewing US imaging with a ring array configuration has the potential to be a viable alternative to conventional US image-guided needle insertion methods.

Modular robotic platform for precision neurosurgery with a bio-inspired needle: System overview and first in-vivo deployment

Over the past 10 years, minimally invasive surgery (MIS) has shown significant benefits compared to conventional surgical techniques, with reduced trauma, shorter hospital stays, and shorter patient recovery times. In neurosurgical MIS procedures, inserting a straight tool (e.g. catheter) is common practice in applications ranging from biopsy and laser ablation, to drug delivery and fluid evacuation. How to handle tissue deformation, target migration and access to deep-seated anatomical structures remain an open challenge, affecting both the preoperative planning phase and eventual surgical intervention. Here, we present the first neurosurgical platform in the literature, able to deliver an implantable steerable needle for a range of diagnostic and therapeutic applications, with a short-term focus on localised drug delivery. This work presents the system's architecture and first in vivo deployment with an optimised surgical workflow designed for pre-clinical trials with the ovine model, which demonstrate appropriate function and safe implantation.

Needle detection using ultrasound B-mode and power Doppler analyses

BACKGROUND

Ultrasound is employed in needle interventions to visualize the anatomical structures and track the needle. Nevertheless, needle detection in ultrasound images is a difficult task, specifically at steep insertion angles.

PURPOSE

A new method is presented to enable effective needle detection using ultrasound B-mode and power Doppler analyses.

METHODS

A small buzzer is used to excite the needle and an ultrasound system is utilized to acquire B-mode and power Doppler images for the needle. The B-mode and power Doppler images are processed using Radon transform and local phase analysis to initially detect the axis of the needle. The detection of the needle axis is improved by processing the power Doppler image using alpha shape analysis to define a region of interest (ROI) that contains the needle. Also, a set of feature maps are extracted from the ROI in the B-mode image. The feature maps are processed using a machine learning classifier to construct a likelihood image that visualizes the posterior needle likelihoods of the pixels. Radon transform is applied to the likelihood image to achieve an improved needle axis detection. Additionally, the region in the B-mode image surrounding the needle axis is analyzed to identify the needle tip using a custom-made probabilistic approach. Our method was utilized to detect needles inserted in ex vivo animal tissues at shallow [20° -40°), moderate [40° -60°), and steep [60° -85°] angles.

RESULTS

Our method detected the needles with failure rates equal to 0% and mean angle, axis, and tip errors less than or equal to 0.7°, 0.6 mm, and 0.7 mm, respectively. Additionally, our method achieved favorable results compared to two recently introduced needle detection methods.

CONCLUSIONS

The results indicate the potential of applying our method to achieve effective needle detection in ultrasound images. This article is protected by copyright. All rights reserved.

Real-Time Ultrasound Doppler Tracking and Autonomous Navigation of a Miniature Helical Robot for Accelerating Thrombolysis in Dynamic Blood Flow

Untethered small-scale robots offer great promise for medical applications in complex biological environments. However, challenges remain in the control and medical imaging of a robot for targeted delivery inside a living body, especially in flowing conditions (e.g., blood vessels). In this work, we report a strategy to autonomously navigate a miniature helical robot in dynamic blood flow under ultrasound Doppler imaging guidance. A magnetic torque and force-hybrid control approach is implemented, enabling the actuation of a millimeter-scale helical robot against blood flow under a rotating magnetic field with a controllable field gradient. Experimental results demonstrate that the robot (length 7.30 mm; diameter 2.15 mm) exhibits controlled navigation in vascular environments, including upstream and downstream navigation in flowing and pulsatile flowing blood with flow rates up to 24 mL/min (mean flow velocity: 14.15 mm/s). During navigation, the rotating robot-induced Doppler signals enable real-time localization and tracking in flowing and pulsatile flowing blood environments. Moreover, the robot can be selectively navigated along different paths by actively controlling the robot's orientation. We apply this autonomous strategy for localizing thrombus and accelerating thrombolysis rate. Compared with conventional tissue plasminogen activator (tPA) thrombolysis, the robot-enhanced shear stress and tPA convection near the clot-blood interface increase the unblocking and thrombolysis efficiency up to 4.8- and 3.5-fold, respectively. Such a medical imaging-guided navigation strategy provides simultaneous robot navigation and localization in complex dynamic biological environments, providing an intelligent approach toward real-time targeted delivery and diagnostic applications in vivo.

Preoperative Motion Planner for Steerable Needles Using Cost Map Based on Repulsive Field and Empirical Model of Needle Deflection

Needle insertion is a common procedure in percutaneous puncture. A motion planner for a steerable needle that considers the risk level of the path in anatomical environment and the actual deflection of clinical needle is necessary. A novel preoperative motion planner for a steerable needle controlled by robot is proposed. Our method utilizes sampling-based planner to compute candidate path in the reachable region, the path solutions are optimized by calculating the cost of a path based on a cost map. The cost-map, which is built based on repulsive field theory from CT image, encodes the information of the obstacle locations and the criticality of the anatomical environment. The empirical formula that can predict needle trajectory is obtained by insertion experiments. Experiments shown that positioning error in gelatin phantom under the guidance of our planner is less than 1.1mm. Comparing with the straight-line insertion method, the positioning error was reduced by 80%. The results indicate that the motion planner has the potential to provide effective guidance for robot-assisted puncture surgery while enhancing the position precision and patient safety.

Coordinated control of a 3DOF cartesian robot and a shape memory alloy-actuated flexible needle for surgical interventions: a non-model-based control method

Success of any needle-based medical procedures depends on accurate placement of the needle at the target location. However, accurate targeting and control of flexible self-actuating (active) needle are challenging. We have developed a shape memory alloy-actuated flexible needle steered by a 3D Cartesian robot and performed a comparative study of four, non-model-based, coordinated control methodologies for the combined robot steering and flexible-needle insertion process for surgical interventions. Investigated four controllers are: proportional–integral–derivative (PID), PID with the cubic of positional error term (PID-P3), static PID sliding mode controller, and robust adaptive PID sliding mode controller (RAPID-SMC). Relative efficacies of these controllers are demonstrated by performing experiements using a tissue-mimicking soft material phantom. Results from experiments have reavealed that RAPID-SMC is superior to other three controllers.

Integrating robot-assisted ultrasound tracking and 3D needle shape prediction for real-time tracking of the needle tip in needle steering procedures

BACKGROUND

Needle insertions have been used in several minimally invasive procedures for diagnostic and therapeutic purposes. Real-time position of the needle tip is an important information in needle steering systems.

METHODS

This work introduces a robot-assisted ultrasound tracking (R-AUST) system integrated with a needle shape prediction method to provide 3D position of the needle tip. The tracking system is evaluated in phantom and ex vivo beef liver tissues.

RESULTS

An average error of 0.60 mm was found for needle insertion tests inside the phantom tissue. The R-AUST integrated with shape prediction in the beef liver tissue was able to track the needle tip with an average and maximum error of 0.37 and 0.67 mm, respectively. The average error reported in this work is within the mean allowable needle placement error (<2.7 mm) in targeted procedures.

CONCLUSIONS

Integration of R-AUST tracking method with needle shape prediction results in a reasonably accurate real-time tracking suitable for ultrasound-guided needle insertions.

Scrubbing needles: a simple and costless technique to improve needle tip visibility during US-guided liver interventions

AIMS

To evaluate the echogenicity of a commercially available needle, modified on the tip, by comparing two groups of patients undergoing to percutaneous biliary drainage.

METHODS

In this retrospective analysis 16 percutaneous transhepatic biliary drainage (PTBD) procedures performed on 16 oncologic patients were evaluated. Patients were randomly divided into two groups of eight subjects each; in the first group, a standard needle was adopted (group A); in the second group, the needle was manually modified to create a rough surface (group B), by scrubbing the tip with an 11 scalpel blade for 150 s all around its surface. To objectively quantify US needle tip visibility, the contrast-to-noise ratio (CNR) was calculated analyzing B-mode images by positioning region of interests in correspondence of needle tip and liver parenchyma.

RESULTS

Needle tip echogenicity was significantly higher in group B where the needle tip was modified compared to control group A (p value = 0.014). CNR, considered to objectively evaluate differences among needle tip echogenicity, was significantly higher in group B with respect to control group A (p value = 0.018).

CONCLUSIONS

The proposed method, scrubbing a 22 gauge commercially available needle tip with a scalpel blade, represents an effective technique to improve needle visibility during US-guided punctures of the liver.

Decoupling Steerability from Diameter: Helical Dovetail Laser Patterning for Steerable Needles

The maximum curvature of a steerable needle in soft tissue is highly sensitive to needle shaft stiffness, which has motivated use of small diameter needles in the past. However, desired needle payloads constrain minimum shaft diameters, and shearing along the needle shaft can occur at small diameters and high curvatures. We provide a new way to adjust needle shaft stiffness (thereby enhancing maximum curvature, i.e. "steerability") at diameters selected based on needle payload requirements. We propose helical dovetail laser patterning to increase needle steerability without reducing shaft diameter. Experiments in phantoms and ex vivo animal muscle, brain, liver, and inflated lung tissues demonstrate high steerability in soft tissues. These experiments use needle diameters suitable for various clinical scenarios, and which have been previously limited by steering challenges without helical dovetail patterning. We show that steerable needle targeting remains accurate with established controllers and demonstrate interventional payload delivery (brachytherapy seeds and radiofrequency ablation) through the needle. Helical dovetail patterning decouples steerability from diameter in needle design. It enables diameter to be selected based on clinical requirements rather than being carefully tuned to tissue properties. These results pave the way for new sensors and interventional tools to be integrated into high-curvature steerable needles.

Medical Imaging of Microrobots: Toward In Vivo Applications

Medical microrobots (MRs) have been demonstrated for a variety of non-invasive biomedical applications, such as tissue engineering, drug delivery, and assisted fertilization, among others. However, most of these demonstrations have been carried out in in vitro settings and under optical microscopy, being significantly different from the clinical practice. Thus, medical imaging techniques are required for localizing and tracking such tiny therapeutic machines when used in medical-relevant applications. This review aims at analyzing the state of the art of microrobots imaging by critically discussing the potentialities and limitations of the techniques employed in this field. Moreover, the physics and the working principle behind each analyzed imaging strategy, the spatiotemporal resolution, and the penetration depth are thoroughly discussed. The paper deals with the suitability of each imaging technique for tracking single or swarms of MRs and discusses the scenarios where contrast or imaging agent's inclusion is required, either to absorb, emit, or reflect a determined physical signal detected by an external system. Finally, the review highlights the existing challenges and perspective solutions which could be promising for future in vivo applications.

Optical Coherence Tomography-Guided Robotic Ophthalmic Microsurgery via Reinforcement Learning from Demonstration

Ophthalmic microsurgery is technically difficult because the scale of required surgical tool manipulations challenge the limits of the surgeon's visual acuity, sensory perception, and physical dexterity. Intraoperative optical coherence tomography (OCT) imaging with micrometer-scale resolution is increasingly being used to monitor and provide enhanced real-time visualization of ophthalmic surgical maneuvers, but surgeons still face physical limitations when manipulating instruments inside the eye. Autonomously controlled robots are one avenue for overcoming these physical limitations. We demonstrate the feasibility of using learning from demonstration and reinforcement learning with an industrial robot to perform OCT-guided corneal needle insertions in an ex vivo model of deep anterior lamellar keratoplasty (DALK) surgery. Our reinforcement learning agent trained on ex vivo human corneas, then outperformed surgical fellows in reaching a target needle insertion depth in mock corneal surgery trials. This work shows the combination of learning from demonstration and reinforcement learning is a viable option for performing OCT guided robotic ophthalmic surgery.

Developing a novel force forecasting technique for early prediction of critical events in robotics

Safety critical events in robotic applications can often be characterized by forces between the robot end-effector and the environment. One application in which safe interaction between the robot and environment is critical is in the area of medical robots. In this paper, we propose a novel Compact Form Dynamic Linearization Model-Free Prediction (CFDL-MFP) technique to predict future values of any time-series sensor data, such as interaction forces. Existing time series forecasting methods have high computational times which motivates the development of a novel technique. Using Autoregressive Integrated Moving Average (ARIMA) forecasting as benchmark, the performance of the proposed model was evaluated in terms of accuracy, computation efficiency, and stability on various force profiles. The proposed algorithm was 11% more accurate than ARIMA and maximum computation time of CFDL-MFP was 4ms, compared to ARIMA (7390ms). Furthermore, we evaluate the model in the special case of predicting needle buckling events, before they occur, by using only axial force and needle-tip position data. The model was evaluated experimentally for robustness with steerable needle insertions into different tissues including gelatin and biological tissue. For a needle insertion velocity of 2.5mm/s, the proposed algorithm was able to predict needle buckling 2.03s sooner than human detections. In biological tissue, no false positive or false negative buckling detections occurred and the rates were low in artificial tissue. The proposed forecasting model can be used to ensure safe robot interactions with delicate environments by predicting adverse force-based events before they occur.

Reliable and accurate needle localization in curvilinear ultrasound images using signature-based analysis of ultrasound beamformed radio frequency signals

PURPOSE

Ultrasound imaging is used in many minimally invasive needle insertion procedures to track the advancing needle, but localizing the needle in ultrasound images can be challenging, particularly at steep insertion angles. Previous methods have been introduced to localize the needle in ultrasound images, but the majority of these methods are based on ultrasound B-mode image analysis that is affected by the needle visibility. To address this limitation, we propose a two-phase, signature-based method to achieve reliable and accurate needle localization in curvilinear ultrasound images based on the beamformed radio frequency (RF) signals that are acquired using conventional ultrasound imaging systems.

METHODS

In the first phase of our proposed method, the beamformed RF signals are divided into overlapping segments and these segments are processed to extract needle-specific features to identify the needle echoes. The features are analyzed using a support vector machine classifier to synthesize a quantitative image that highlights the needle. The quantitative image is processed using the Radon transform to achieve a reliable and accurate signature-based estimation of the needle axis. In the second phase, the accuracy of the needle axis estimation is improved by processing the RF samples located around the signature-based estimation of the needle axis using local phase analysis combined with the Radon transform. Moreover, a probabilistic approach is employed to identify the needle tip. The proposed method is used to localize needles with two different sizes inserted in ex vivo animal tissue specimens at various insertion angles.

RESULTS

Our proposed method achieved reliable and accurate needle localization for an extended range of needle insertion angles with failure rates of 0% and mean angle, axis, and tip errors smaller than or equal to 0 . 7 ∘ , 0.6 mm, and 0.7 mm, respectively. Moreover, our proposed method outperformed a recently introduced needle localization method that is based on B-mode image analysis.

CONCLUSIONS

These results suggest the potential of employing our signature-based method to achieve reliable and accurate needle localization during ultrasound-guided needle insertion procedures.

Magnetic Needle Steering in Soft Phantom Tissue

Needle steering is a technology for guiding needles around sensitive internal obstacles in minimally invasive surgery. Traditional techniques apply rotation at the base of a needle with an asymmetric tip, enabling steering through the redirection of radial forces. Magnetic steering of catheters and continuum manipulators is another technology that allows steering of a shaft in the body. Both of these techniques rely on mechanical or manual shaft advancement methods. Needle steering has not achieved widespread clinical use due to several limitations: 1- buckling and compression effects in the shaft and needle rotation cause excessive tissue damage; 2- torsion effects on the shaft and needle deflection at tissue boundaries lead to difficulty in control; and 3- restricted radius of curvature results in limited workspace. Magnetically steered catheters and continuum manipulators also suffer from limited curvature and the possibility of buckling. This paper proposes a novel needle steering method empowered by electromagnetic actuation that overcomes all of the aforementioned limitations, making it a promising option for further study toward healthcare applications.

Ring-arrayed Forward-viewing Ultrasound Imaging System: A Feasibility Study

Current standard workflows of ultrasound (US)-guided needle insertion require physicians to use their both hands: holding the US probe to locate interested areas with the non-dominant hand and the needle with the dominant hand. This is due to the separation of functionalities for localization and needle insertion. This requirement does not only make the procedure cumbersome, but also limits the reliability of guidance given that the positional relationship between the needle and US images is unknown and interpreted with their experience and assumption. Although the US-guided needle insertion may be assisted through navigation systems, recovery of the positional relationship between the needle and US images requires the usage of external tracking systems and image-based tracking algorisms that may involve the registration inaccuracy. Therefore, there is an unmet need for the solution that provides a simple and intuitive needle localization and insertion to improve the conventional US-guided procedure. In this work, we propose a new device concept solution based on the ring-arrayed forward-viewing (RAF) ultrasound imaging system. The proposed system is comprised with ring-arrayed transducers and an open whole inside the ring where the needle can be inserted. The ring array provides forward-viewing US images, where the needle path is always maintained at the center of the reconstructed image without requiring any registration. As the proof of concept, we designed single-circle ring-arrayed configurations with different radiuses and visualized point targets using the forward-viewing US imaging through simulations and phantom experiments. The results demonstrated the successful target visualization and indicates the ring-arrayed US imaging has a potential to improve the US-guided needle insertion procedure to be simpler and more intuitive.

Steerable Microinvasive Probes for Localized Drug Delivery to Deep Tissue

Enhanced understanding of neuropathologies has created a need for more advanced tools. Current neural implants result in extensive glial scarring and are not able to highly localize drug delivery due to their size. Smaller implants reduce surgical trauma and improve spatial resolution, but such a reduction requires improvements in device design to enable accurate and chronic implantation in subcortical structures. Flexible needle steering techniques offer improved control over implant placement, but often require complex closed-loop control for accurate implantation. This study reports the development of steerable microinvasive neural implants (S-MINIs) constructed from borosilicate capillaries (OD = 60 µm, ID = 20 µm) that do not require closed-loop guidance or guide tubes. S-MINIs reduce glial scarring 3.5-fold compared to prior implants. Bevel steered needles are utilized for open-loop targeting of deep-brain structures. This study demonstrates a sinusoidal relationship between implant bevel angle and the trajectory radius of curvature both in vitro and ex vivo. This relationship allows for bevel-tipped capillaries to be steered to a target with an average error of 0.23 mm ± 0.19 without closed-loop control. Polished microcapillaries present a new microinvasive tool for chronic, predictable targeting of pathophysiological structures without the need for closed-loop feedback and complex imaging.

Enhanced needle visibility by microbubbles generated with negative pressure using an in-plane technique

BACKGROUND AND OBJECTIVES

Our previous work found that needle visibility could be improved by introducing microbubbles into needles. The primary aim of this study was to test the hypothesis that the negative pressure method is superior to two other methods for enhancing needle visibility by introducing microbubbles into needles. The secondary aim was to evaluate the impacts of three factors on the visibility of microbubble-filled needles.

METHODS

In the first phase, three methods, including the negative pressure method, the mixing method and commercialized microbubbles, were applied to generate microbubbles inside needles for comparison of visibility in a porcine meat model. In the second phase, three factors were tested with a 2×5×5 factorial design to explore their influence on the visibility of microbubble-filled needles. The three factors included types of needles, insertion angles and types of contents inside needles. Needles filled with saline without microbubbles were used as the control in both phases. Insertion videos were recorded, and ultrasound images of needles were captured for the objective visibility analysis.

RESULTS

Needle visibility was highest in the negative pressure method group (p<0.001). Needle visibility was mainly determined by insertion angles (p<0.001). Microbubble-filled needles were more visible than control needles at 40°, 50° and 60° (p<0.001, p<0.001 and p<0.001, respectively).

CONCLUSIONS

Needle visibility can be significantly improved by microbubbles generated with the negative pressure method when insertion angles are 40°, 50° and 60° in porcine meat.

Three-dimensional therapy needle applicator segmentation for ultrasound-guided focal liver ablation

PURPOSE

Minimally invasive procedures, such as microwave ablation, are becoming first-line treatment options for early-stage liver cancer due to lower complication rates and shorter recovery times than conventional surgical techniques. Although these procedures are promising, one reason preventing widespread adoption is inadequate local tumor ablation leading to observations of higher local cancer recurrence compared to conventional procedures. Poor ablation coverage has been associated with two-dimensional (2D) ultrasound (US) guidance of the therapy needle applicators and has stimulated investigation into the use of three-dimensional (3D) US imaging for these procedures. We have developed a supervised 3D US needle applicator segmentation algorithm using a single user input to augment the addition of 3D US to the current focal liver tumor ablation workflow with the goals of identifying and improving needle applicator localization efficiency.

METHODS

The algorithm is initialized by creating a spherical search space of line segments around a manually chosen seed point that is selected by a user on the needle applicator visualized in a 3D US image. The most probable trajectory is chosen by maximizing the count and intensity of threshold voxels along a line segment and is filtered using the Otsu method to determine the tip location. Homogeneous tissue mimicking phantom images containing needle applicators were used to optimize the parameters of the algorithm prior to a four-user investigation on retrospective 3D US images of patients who underwent microwave ablation for liver cancer. Trajectory, axis localization, and tip errors were computed based on comparisons to manual segmentations in 3D US images.

RESULTS

Segmentation of needle applicators in ten phantom 3D US images was optimized to median (Q1, Q3) trajectory, axis, and tip errors of 2.1 (1.1, 3.6)°, 1.3 (0.8, 2.1) mm, and 1.3 (0.7, 2.5) mm, respectively, with a mean ± SD segmentation computation time of 0.246 ± 0.007 s. Use of the segmentation method with a 16 in vivo 3D US patient dataset resulted in median (Q1, Q3) trajectory, axis, and tip errors of 4.5 (2.4, 5.2)°, 1.9 (1.7, 2.1) mm, and 5.1 (2.2, 5.9) mm based on all users.

CONCLUSIONS

Segmentation of needle applicators in 3D US images during minimally invasive liver cancer therapeutic procedures could provide a utility that enables enhanced needle applicator guidance, placement verification, and improved clinical workflow. A semi-automated 3D US needle applicator segmentation algorithm used in vivo demonstrated localization of the visualized trajectory and tip with less than 5° and 5.2 mm errors, respectively, in less than 0.31 s. This offers the ability to assess and adjust needle applicator placements intraoperatively to potentially decrease the observed liver cancer recurrence rates associated with current ablation procedures. Although optimized for deep and oblique angle needle applicator insertions, this proposed workflow has the potential to be altered for a variety of image-guided minimally invasive procedures to improve localization and verification of therapy needle applicators intraoperatively.

A Methodical Quantification of Needle Visibility and Echogenicity in Ultrasound Images

During ultrasound-guided percutaneous interventions, needle localization can be a challenge. To increase needle visibility, enhancements of both the imaging methods and the needle surface properties have been investigated. However, a methodical approach to compare potential solutions is currently unavailable. The work described here involves automated image acquisition, analysis and reporting techniques to collect large amounts of data efficiently, delineate relevant factors and communicate effects. Data processing included filtering, line fitting and image intensity analysis steps. Foreground and background image samples were used to compute a contrast-to-noise ratio or a signal ratio. The approach was evaluated in a comparative study of commercially available and custom-made needles. Varied parameters included needle material, diameter and surface roughness. The shafts with kerfed patterns and the trocar and chiba tips performed best. The approach enabled an intuitive polar depiction of needle visibility in ultrasound images for a large range of insertion angles.

Accurate Needle Localization Using Two-Dimensional Power Doppler and B-Mode Ultrasound Image Analyses: A Feasibility Study

Curvilinear ultrasound transducers are commonly used in various needle insertion interventions, but localizing the needle in curvilinear ultrasound images is usually challenging. In this paper, a new method is proposed to localize the needle in curvilinear ultrasound images by exciting the needle using a piezoelectric buzzer and imaging the excited needle using a curvilinear ultrasound transducer to acquire a power Doppler image and a B-mode image. The needle-induced Doppler responses that appear in the power Doppler image are analyzed to estimate the needle axis initially and identify the candidate regions that are expected to include the needle. The candidate needle regions in the B-mode image are analyzed to improve the localization of the needle axis. The needle tip is determined by analyzing the intensity variations of the power Doppler and B-mode images around the needle axis. The proposed method is employed to localize different needles that are inserted in three ex vivo animal tissue types at various insertion angles, and the results demonstrate the capability of the method to achieve automatic, reliable and accurate needle localization. Furthermore, the proposed method outperformed two existing needle localization methods.

Closed-loop control of bevel-tip needles based on path planning

Bevel-tip needles have the potential to improve paracentetic precision and decrease paracentetic traumas. In order to drive bevel-tip needles precisely with the constrains of path length and path dangerousness, we propose a closed-loop control method that only requires the position of the needle tip and can be easily applied in a clinical setting. The control method is based on the path planning method proposed in this paper. To establish the closed-loop control method, a kinematic model of bevel-tip needles is first presented, and the relationship between the puncture path and controlled variables is established. Second, we transform the path planning method into a multi-objective optimization problem, which takes the path error, path length and path dangerousness into account. Multi-objective particle swarm optimization is employed to solve the optimization problem. Then, a control method based on path planning is presented. The current needle tip attitude is essential to plan an insertion path. We analyze two methods to obtain the tip attitude and compare their effects using both simulations and experiments. In the end, simulations and experiments in phantom tissue are executed and analyzed, the results show that our methods have high accuracy and have the ability to deal with the model parameter uncertainty.

Improved Performance of -Mode Needle-Actuating Transducer With PMN-PT Piezocrystal

Prototypes of a PZT-based ultrasound needle-actuating device have shown the ability to reduce needle penetration force and enhance needle visibility with color Doppler imaging during needle insertion for tissue biopsy and regional anesthesia. However, the demand for smaller, lighter devices and the need for high performance transducers have motivated investigation of a different configuration of needle-actuation transducer, utilizing the -mode of PZT4 piezoceramic, and exploration of further improvement in its performance using relaxor-type piezocrystal. This paper outlines the development of the -mode needle actuation transducer design from simulation to fabrication and demonstration. Full characterization was performed on transducers for performance comparison. The performance of the proposed smaller, lighter -mode transducer is comparable with that of previous -mode transducers. Furthermore, it has been found to be much more efficient when using PMN-PT piezocrystal rather than piezoceramic.

Automatic Robotic Steering of Flexible Needles from 3D Ultrasound Images in Phantoms and Ex Vivo Biological Tissue

Robotic control of needle bending aims at increasing the precision of percutaneous procedures. Ultrasound feedback is preferable for its clinical ease of use, cost and compactness but raises needle detection issues. In this paper, we propose a complete system dedicated to robotized guidance of a flexible needle under 3D ultrasound imaging. This system includes a medical robot dedicated to transperineal needle positioning and insertion, a rapid path planning for needle steering using bevel-tip needle natural curvature in tissue, and an ultrasound-based automatic needle detection algorithm. Since ultrasound-based automatic needle steering is often made difficult by the needle localization in biological tissue, we quantify the benefit of using flexible echogenic needles for robotized guidance under 3D ultrasound. The "echogenic" term refers to the etching of microstructures on the needle shaft. We prove that these structures improve needle visibility and detection robustness in ultrasound images. We finally present promising results when reaching targets using needle steering. The experiments were conducted with various needles in different media (synthetic phantoms and ex vivo biological tissue). For instance, with nitinol needles the mean accuracy is 1.2 mm (respectively 3.8 mm) in phantoms (resp. biological tissue).

Laser Doppler sensing for blood vessel detection with a biologically inspired steerable needle

Puncturing blood vessels during percutaneous intervention in minimally invasive brain surgery can be a life threatening complication. Embedding a forward looking sensor in a rigid needle has been proposed to tackle this problem but, when using a rigid needle, the procedure needs to be interrupted and the needle extracted if a vessel is detected. As an alternative, we propose a novel optical method to detect a vessel in front of a steerable needle. The needle itself is based on a biomimetic, multi-segment design featuring four hollow working channels. Initially, a laser Doppler flowmetry probe is characterized in a tissue phantom with optical properties mimicking those of human gray matter. Experiments are performed to show that the probe has a 2.1 mm penetration depth and a 1 mm off-axis detection range for a blood vessel phantom with 5 mm s^-1 flow velocity. This outcome demonstrates that the probe fulfills the minimum requirements for it to be used in conjunction with our needle. A pair of Doppler probes is then embedded in two of the four working channels of the needle and vessel reconstruction is performed using successive measurements to determine the depth and the off-axis position of the vessel from each laser Doppler probe. The off-axis position from each Doppler probe is then used to generate a 'detection circle' per probe, and vessel orientation is predicted using tangent lines between the two. The vessel reconstruction has a depth root mean square error (RMSE) of 0.3 mm and an RMSE of 15° in the angular prediction, showing real promise for a future clinical application of this detection system.

A computationally efficient method for hand-eye calibration

Purpose

Surgical robots with cooperative control and semiautonomous features have shown increasing clinical potential, particularly for repetitive tasks under imaging and vision guidance. Effective performance of an autonomous task requires accurate hand–eye calibration so that the transformation between the robot coordinate frame and the camera coordinates is well defined. In practice, due to changes in surgical instruments, online hand–eye calibration must be performed regularly. In order to ensure seamless execution of the surgical procedure without affecting the normal surgical workflow, it is important to derive fast and efficient hand–eye calibration methods.

Methods

We present a computationally efficient iterative method for hand–eye calibration. In this method, dual quaternion is introduced to represent the rigid transformation, and a two-step iterative method is proposed to recover the real and dual parts of the dual quaternion simultaneously, and thus the estimation of rotation and translation of the transformation.

Results

The proposed method was applied to determine the rigid transformation between the stereo laparoscope and the robot manipulator. Promising experimental and simulation results have shown significant convergence speed improvement to 3 iterations from larger than 30 with regard to standard optimization method, which illustrates the effectiveness and efficiency of the proposed method.

Detection of an invisible needle in ultrasound using a probabilistic SVM and time-domain features

We propose a novel learning-based approach to detect an imperceptible hand-held needle in ultrasound images using the natural tremor motion. The minute tremor induced on the needle however is also transferred to the tissue in contact with the needle, making the accurate needle detection a challenging task. The proposed learning-based framework is based on temporal analysis of the phase variations of pixels to classify them according to the motion characteristics. In addition to the classification, we also obtain a probability map of the segmented pixels by cross-validation. A Hough transform is then used on the probability map to localize the needle using the segmented needle and posterior probability estimate. The two-step probability-weighted localization on the segmented needle in a learning framework is the key innovation which results in localization improvement and adaptability to specific clinical applications. The method was tested in vivo for a standard 17 gauge needle inserted at 50-80° insertion angles and 40-60mm depths. The results showed an average accuracy of (2.12°, 1.69mm) and 81%±4% for localization and classification, respectively.

CASPER: computer-aided segmentation of imperceptible motion-a learning-based tracking of an invisible needle in ultrasound

PURPOSE

This paper presents a new micro-motion-based approach to track a needle in ultrasound images captured by a handheld transducer.

METHODS

We propose a novel learning-based framework to track a handheld needle by detecting microscale variations of motion dynamics over time. The current state of the art on using motion analysis for needle detection uses absolute motion and hence work well only when the transducer is static. We have introduced and evaluated novel spatiotemporal and spectral features, obtained from the phase image, in a self-supervised tracking framework to improve the detection accuracy in the subsequent frames using incremental training. Our proposed tracking method involves volumetric feature selection and differential flow analysis to incorporate the neighboring pixels and mitigate the effects of the subtle tremor motion of a handheld transducer. To evaluate the detection accuracy, the method is tested on porcine tissue in-vivo, during the needle insertion in the biceps femoris muscle.

RESULTS

Experimental results show the mean, standard deviation and root-mean-square errors of [Formula: see text], [Formula: see text] and [Formula: see text] in the insertion angle, and 0.82, 1.21, 1.47 mm, in the needle tip, respectively.

CONCLUSIONS

Compared to the appearance-based detection approaches, the proposed method is especially suitable for needles with ultrasonic characteristics that are imperceptible in the static image and to the naked eye.