Experimental evaluation of magnified haptic feedback for robot-assisted needle insertion and palpation

Virtual Needle Insertion with Enhanced Haptic Feedback for Guidance and Needle-Tissue Interaction Forces

Interventional radiologists mainly rely on visual feedback via imaging modalities to steer a needle toward a tumor during biopsy and ablation procedures. In the case of CT-guided procedures, there is a risk of exposure to hazardous X-ray-based ionizing radiation. Therefore, CT scans are usually not used continuously, which increases the chances of a misplacement of the needle and the need for reinsertion, leading to more tissue trauma. Interventionalists also encounter haptic feedback via needle-tissue interaction forces while steering a needle. These forces are useful but insufficient to clearly perceive and identify deep-tissue structures such as tumors. The objective of this paper was to investigate the effect of enhanced force feedback for sensing interaction forces and guiding the needle when applied individually and simultaneously during a virtual CT-guided needle insertion task. We also compared the enhanced haptic feedback to enhanced visual feedback. We hypothesized that enhancing the haptic feedback limits the time needed to reach the target accurately and reduces the number of CT scans, as the interventionalist depends more on real-time enhanced haptic feedback. To test the hypothesis, a simulation environment was developed to virtually steer a needle in five degrees of freedom (DoF) to reach a tumor target embedded in a liver model. Twelve participants performed in the experiment with different feedback conditions where we measured their performance in terms of the following: targeting accuracy, trajectory tracking, number of CT scans required, and the time needed to finish the task. The results suggest that the combination of enhanced haptic feedback for guidance and sensing needle-tissue interaction forces significantly reduce the number of scans and the duration required to finish the task by 32.1% and 46.9%, respectively, when compared to nonenhanced haptic feedback. The other feedback modalities significantly reduced the duration to finish the task by around 30% compared to nonenhanced haptic feedback.

A Mechanical Evaluation of a Robot-Assisted Cutting Cornea Based on Force Response

The aim of this paper is to propose laws of trephine operation based on a robot-assisted cutting cornea in order to obtain better microsurgical effects for keratoplasty. Using a trephine robot integrated with a microforce sensor and a handheld trephine manipulator, robotic and manual experiments were performed, with porcine corneas as the test subjects. The effect of trephine operational parameters on the results reflected by the biomechanical response is discussed, and the parameters include linear velocity, rotating angle, and angular velocity. Using probability density functions, the distributions of the manual operational parameters show some randomness, and there is a large fluctuation in the trephine force during the experiments. The biomechanical response shows regular trends in the robotic experiments even under different parameters, and compared to manual trephination, the robot may perform the operation of trephine cornea cutting more stably. Under different operational parameters, the cutting force shows different trends, and the optimal initial parameters that result in better trephine effects can be obtained based on the trends. Based on this derived law, the operational parameters can be set in robotic trephination, and surgeons can also be specially trained to achieve a better microsurgical result.

AI enhanced collaborative human-machine interactions for home-based telerehabilitation

The use of robots in a telerehabilitation paradigm could facilitate the delivery of rehabilitation on demand while reducing transportation time and cost. As a result, it helps to motivate patients to exercise frequently in a more comfortable home environment. However, for such a paradigm to work, it is essential that the robustness of the system is not compromised due to network latency, jitter, and delay of the internet. This paper proposes a solution to data loss compensation to maintain the quality of the interaction between the user and the system. Data collected from a well-defined collaborative task using a virtual reality (VR) environment was used to train a robotic system to adapt to the users’ behaviour. The proposed approach uses nonlinear autoregressive models with exogenous input (NARX) and long-short term memory (LSTM) neural networks to smooth out the interaction between the user and the predicted movements generated from the system. LSTM neural networks are shown to learn to act like an actual human. The results from this paper have shown that, with an appropriate training method, the artificial predictor can perform very well by allowing the predictor to complete the task within 25 s versus 23 s when executed by the human.

Force-Sensorless Identification and Classification of Tissue Biomechanical Parameters for Robot-Assisted Palpation

The implementation of robotic systems for minimally invasive surgery and medical procedures is an active topic of research in recent years. One of the most common procedures is the palpation of soft tissues to identify their mechanical characteristics. In particular, it is very useful to identify the tissue's stiffness or equivalently its elasticity coefficient. However, this identification relies on the existence of a force sensor or a tactile sensor mounted at the tip of the robot, as well as on measuring the robot velocity. For some applications it would be desirable to identify the biomechanical characteristics of soft tissues without the need for a force/tactile nor velocity sensors. An estimation of such quantities can be obtained by a model-based state observer for which the inputs are only the robot joint positions and its commanded joint torques. The estimated velocities and forces can then be employed for closed-loop force control, force reflection, and mechanical parameters estimation. In this work, a closed-loop force control is proposed based on the estimated contact forces to avoid any tissue damage. Then, the information from the estimated forces and velocities is used in a least squares estimator of the mechanical parameters. Moreover, the estimated biomechanical parameters are employed in a Bayesian classifier to provide further help for the physician to make a diagnosis. We have found that a combination of the parameters of both linear and nonlinear viscoelastic models provide better classification results: 0% misclassifications against 50% when using a linear model, and 3.12% when using only a nonlinear model, for the case in which the samples have very similar mechanical properties.

Experimental Evaluation on Haptic Feedback Accuracy by Using Two Self-Made Haptic Devices and One Additional Interface in Robotic Teleoperation

The goal of haptic feedback in robotic teleoperation is to enable users to accurately feel the interaction force measured at the slave side and precisely understand what is happening in the slave environment. The accuracy of the feedback force describing the error between the actual feedback force felt by a user at the master side and the measured interaction force at the slave side is the key performance indicator for haptic display in robotic teleoperation. In this paper, we evaluate the haptic feedback accuracy in robotic teleoperation via experimental method. A special interface iHandle and two haptic devices, iGrasp-T and iGrasp-R, designed for robotic teleoperation are developed for experimental evaluation. The device iHandle integrates a high-performance force sensor and a micro attitude and heading reference system which can be used to identify human upper limb motor abilities, such as posture maintenance and force application. When a user is asked to grasp the iHandle and maintain a fixed position and posture, the fluctuation value of hand posture is measured to be between 2 and 8 degrees. Based on the experimental results, human hand tremble as input noise sensed by the haptic device is found to be a major reason that results in the noise of output force from haptic device if the spring-damping model is used to render feedback force. Therefore, haptic rendering algorithms should be independent of hand motion information to avoid input noise from human hand to the haptic control loop in teleoperation. Moreover, the iHandle can be fixed at the end effector of haptic devices; iGrasp-T or iGrasp-R, to measure the output force/torque from iGrasp-T or iGrasp-Rand to the user. Experimental results show that the accuracy of the output force from haptic device iGrasp-T is approximately 0.92 N, and using the force sensor in the iHandle can compensate for the output force inaccuracy of device iGrasp-T to 0.1 N. Using a force sensor as the feedback link to form a closed-loop feedback force control system is an effective way to improve the accuracy of feedback force and guarantee high-fidelity of feedback forces at the master side in robotic teleoperation.

Reducing retraction forces with tactile feedback during robotic total mesorectal excision in a porcine model

Excessive tissue-instrument interaction forces during robotic surgery have the potential for causing iatrogenic tissue damages. The current in vivo study seeks to assess whether tactile feedback could reduce intraoperative tissue-instrument interaction forces during robotic-assisted total mesorectal excision. Five subjects, including three experts and two novices, used the da Vinci robot to perform total mesorectum excision in four pigs. The grip force in the left arm, used for retraction, and the pushing force in the right arm, used for blunt pelvic dissection around the rectum, were recorded. Tissue-instrument interaction forces were compared between trials done with and without tactile feedback. The mean force exerted on the tissue was consistently higher in the retracting arm than the dissecting arm (3.72 ± 1.19 vs 0.32 ± 0.36 N, p < 0.01). Tactile feedback brought about significant reductions in average retraction forces (3.69 ± 1.08 N vs 4.16 ± 1.12 N, p = 0.02), but dissection forces appeared unaffected (0.43 ± 0.42 vs 0.37 ± 0.28 N, p = 0.71). No significant differences were found between retraction and dissection forces exerted by novice and expert robotic surgeons. This in vivo animal study demonstrated the efficacy of tactile feedback in reducing retraction forces during total mesorectal excision. Further research is required to quantify the clinical impact of such force reduction.

Real-Time Compression for Tactile Internet Data Streams

The Tactile Internet will require ultra-low latencies for combining machines and humans in systems where humans are in the control loop. Real-time and perceptual coding in these systems commonly require content-specific approaches. We present a generic approach based on deliberately reduced number accuracy and evaluate the trade-off between savings achieved and errors introduced with real-world data for kinesthetic movement and tele-surgery. Our combination of bitplane-level accuracy adaptability with perceptual threshold-based limits allows for great flexibility in broad application scenarios. Combining the attainable savings with the relatively small introduced errors enables the optimal selection of a working point for the method in actual implementations.

A review of haptic feedback in tele-operated robotic surgery

During traditional surgery, the surgeons' hands are in direct contact with organs, and surgeons rely on the sense of touch to perform surgery. In teleoperated robotic systems, all physical connections between the surgeon and both the robot and patient, are absent. The surgeon must estimate the force exerted on organs, based only on visual deformation of tissues he is pulling, pushing, gripping, or suturing. It is hard to imagine how to operate with no haptic sensations, and it is surprising that commercially available robots didn't include until now any Haptic Feedback, despite reports about tissue injury, and inability to perform complex manipulation. The sense of touch must be created by stimuli sensed by the surgeon. Haptic sensors are required to collect and send haptic information, and display them on the operator's side, creating telepresence, known as transparency. Multiple ways have been developed to improve transparency through force feedback and tactile feedback. However, this interferes with the stability of the closed-loop controlling interactions between master, robot and remote environment. Cutaneous feedback is more stable and less transparent; force feedback is more transparent and less stable. Thus, multimodal platforms of haptic feedback would try to find the best trade-off between both modalities.

Haptics in Teleoperated Medical Interventions: Force Measurement, Haptic Interfaces and Their Influence on User's Performance

OBJECTIVES

Haptics in teleoperated medical interventions enables measurement and transfer of force information to the operator during robot-environment interaction. This paper provides an overview of the current research in this domain and guidelines for future investigations.

METHODS

We review current technologies in force measurement and haptic devices as well as their experimental evaluation and influence on user's performance.

RESULTS

Force sensing is moving away from the conventional proximal measurement methods to distal sensing and contact-less methods. Wearable devices that deliver haptic feedback on different body parts are increasingly playing an important role. Performance and accuracy improvement are the widely reported benefits of haptic feedback, while there is a debate on its effect on task completion time and exerted force.

CONCLUSION

With the surge of new ideas, there is a need for better and more systematic validation of the new sensing and feedback technology, through better user studies and novel methods like validated benchmarks and new taxonomies.

SIGNIFICANCE

This review investigates haptics from sensing to interfaces within the context of user's performance and the validation procedures to highlight salient advances. It provides guidelines to future developments and highlights the shortcomings in the field.

A Haptic Shared-Control Architecture for Guided Multi-Target Robotic Grasping

Although robotic telemanipulation has always been a key technology for the nuclear industry, little advancement has been seen over the last decades. Despite complex remote handling requirements, simple mechanically linked master-slave manipulators still dominate the field. Nonetheless, there is a pressing need for more effective robotic solutions able to significantly speed up the decommissioning of legacy radioactive waste. This paper describes a novel haptic shared-control approach for assisting a human operator in the sort and segregation of different objects in a cluttered and unknown environment. A three-dimensional scan of the scene is used to generate a set of potential grasp candidates on the objects at hand. These grasp candidates are then used to generate guiding haptic cues, which assist the operator in approaching and grasping the objects. The haptic feedback is designed to be smooth and continuous as the user switches from a grasp candidate to the next one, or from one object to another one, avoiding any discontinuity or abrupt changes. To validate our approach, we carried out two human-subject studies, enrolling 15 participants. We registered an average improvement of 20.8%, 20.1%, and 32.5% in terms of completion time, linear trajectory, and perceived effectiveness, respectively, between the proposed approach and standard teleoperation.

Caring About the Human Operator: Haptic Shared Control for Enhanced User Comfort in Robotic Telemanipulation

Haptic shared control enables a human operator and an autonomous controller to share the control of a robotic system using haptic active constraints. It has been used in robotic teleoperation for different purposes, such as navigating along paths minimizing the torques requested to the manipulator or avoiding possibly dangerous areas of the workspace. However, few works have focused on using these ideas to account for the user's comfort. In this article, we present an innovative haptic-enabled shared control approach aimed at minimizing the user's workload during a teleoperated manipulation task. Using an inverse kinematic model of the human arm and the rapid upper limb assessment (RULA) metric, the proposed approach estimates the current user's comfort online. From this measure and an a priori knowledge of the task, we then generate dynamic active constraints guiding the users towards a successful completion of the task, along directions that improve their posture and increase their comfort. Studies with human subjects show the effectiveness of the proposed approach, yielding a 30% perceived reduction of the workload with respect to using standard guided human-in-the-loop teleoperation.

Unifying bilateral teleoperation and tele-impedance for enhanced user experience

Usability is one of the most important aspects of teleoperation. Ideally, the operator’s experience should be one of complete command over the remote environment, but also be as close as possible to what they would have if physically present at the remote end, i.e., transparency in terms of both action and perception. These two aspects may coincide in favorable conditions, where classic approaches such as the four-channel architecture ensures transparency of the control framework. In the presence of substantial delays between the user and the slave, however, the stability–performance trade-off inherent to bilateral teleoperation deteriorates not only transparency, but also command. An alternative, unilateral approach is given by tele-impedance, which controls the slave–environment interaction by measuring and remotely replicating the user’s limb endpoint position and impedance. Not including force feedback to the operator, tele-impedance is absolutely robust to delays, whereas it completely lacks transparency. This article introduces a novel control framework that integrates a new, fully transparent, two-channel bilateral architecture with the tele-impedance paradigm. The result is a unified solution that mitigates problems of classical approaches, and provides the user with additional tools to modulate the slave robot’s physical interaction behavior, resulting in a better operator experience in spite of time inconsistencies. The validity and effectiveness of the proposed solution is demonstrated in terms of performance in the interaction tasks, of user fatigue and overall experience.

Spatio-temporal deep learning models for tip force estimation during needle insertion

PURPOSE

Precise placement of needles is a challenge in a number of clinical applications such as brachytherapy or biopsy. Forces acting at the needle cause tissue deformation and needle deflection which in turn may lead to misplacement or injury. Hence, a number of approaches to estimate the forces at the needle have been proposed. Yet, integrating sensors into the needle tip is challenging and a careful calibration is required to obtain good force estimates.

METHODS

We describe a fiber-optic needle tip force sensor design using a single OCT fiber for measurement. The fiber images the deformation of an epoxy layer placed below the needle tip which results in a stream of 1D depth profiles. We study different deep learning approaches to facilitate calibration between this spatio-temporal image data and the related forces. In particular, we propose a novel convGRU-CNN architecture for simultaneous spatial and temporal data processing.

RESULTS

The needle can be adapted to different operating ranges by changing the stiffness of the epoxy layer. Likewise, calibration can be adapted by training the deep learning models. Our novel convGRU-CNN architecture results in the lowest mean absolute error of [Formula: see text] and a cross-correlation coefficient of 0.9997 and clearly outperforms the other methods. Ex vivo experiments in human prostate tissue demonstrate the needle's application.

CONCLUSIONS

Our OCT-based fiber-optic sensor presents a viable alternative for needle tip force estimation. The results indicate that the rich spatio-temporal information included in the stream of images showing the deformation throughout the epoxy layer can be effectively used by deep learning models. Particularly, we demonstrate that the convGRU-CNN architecture performs favorably, making it a promising approach for other spatio-temporal learning problems.

Haptic feedback in the da Vinci Research Kit (dVRK): A user study based on grasping, palpation, and incision tasks

BACKGROUND

It was suggested that the lack of haptic feedback, formerly considered a limitation for the da Vinci robotic system, does not affect robotic surgeons because of training and compensation based on visual feedback. However, conclusive studies are still missing, and the interest in force reflection is rising again.

METHODS

We integrated a seven-DoF master into the da Vinci Research Kit. We designed tissue grasping, palpation, and incision tasks with robotic surgeons, to be performed by three groups of users (expert surgeons, medical residents, and nonsurgeons, five users/group), either with or without haptic feedback. Task-specific quantitative metrics and a questionnaire were used for assessment.

RESULTS

Force reflection made a statistically significant difference for both palpation (improved inclusion detection rate) and incision (decreased tissue damage).

CONCLUSIONS

Haptic feedback can improve key surgical outcomes for tasks requiring a pronounced cognitive burden for the surgeon, to be possibly negotiated with longer completion times.

Share control of surgery robot master manipulator guiding tool along the standard path

Recently, minimally invasive surgery (MIS) robotics enters the phase of autonomous operation. However, because of the high variability of the environment, conducting a fully autonomous surgery is still extremely difficult. This paper presents a share control system, the objective of which is to suggest the optimum path of tool guidance through the use of force on the master manipulator (hereinafter as master), meaning the surgeon's hand. Owing to this type of control, the surgeon has full control over the position of the tool the entire time and is supported by the system to better and faster guide the tool during surgery. The force should be felt by the surgeon but, simultaneously, must not hinder or impact the surgical process. Furthermore, the share control system presented in the paper can be turned on or off at any moment during surgery.

Evaluation of Haptic Feedback on Bimanually Teleoperated Laparoscopy for Endometriosis Surgery

Robotic minimal invasive surgery is gaining acceptance in surgical care. In contrast with the appreciated three-dimensional vision and enhanced dexterity, haptic feedback is not offered. For this reason, robotics is not considered beneficial for delicate interventions such as the endometriosis. Overall, haptic feedback remains debatable and yet unproven except for some simple scenarios such as fundamentals of laparoscopic surgery exercises. Objective: This work investigates the benefits of haptic feedback on more complex surgical gestures, manipulating delicate tissue through coordination between multiple instruments. Methods: A new training exercise, “endometriosis surgery exercise” (ESE) has been devised approximating the setting for monocular robotic endometriosis treatment. A bimanual bilateral teleoperation setup was designed for laparoscopic laser surgery. Haptic guidance and haptic feedback are, respectively, offered to the operator. User experiments have been conducted to assess the validity of ESE and examine possible advantages of haptic technology during execution of bimanual surgery. Results: Content and face validity of ESE were established by participating surgeons. Surgeons suggested ESE also as a mean to train lasering skills, and interaction forces on endometriotic tissue were found to be significantly lower when a bilateral controller is used. Collisions between instruments and the environment were less frequent and so were situations marked as potentially dangerous. Conclusion: This study provides some promising results suggesting that haptics may offer a distinct advantage in complex robotic interventions were fragile tissue is manipulated. Significance: Patients need to know whether it should be incorporated. Improved understanding of the value of haptics is important as current commercial surgical robots are widely used but do not offer haptics.

DNN-Based Assistant in Laparoscopic Computer-Aided Palpation

Tactile sensory input of surgeons is severely limited in minimally invasive surgery, therefore manual palpation cannot be performed for intraoperative tumor detection. Computer-aided palpation, in which tactile information is acquired by devices and relayed to the surgeon, is one solution for overcoming this limitation. An important design factor is the method by which the acquired information is fed back to the surgeon. However, currently there is no systematic method for achieving this aim, and it is possible that a badly implemented feedback mechanism could adversely affect the performance of the surgeon. In this study, we propose an assistance algorithm for intraoperative tumor detection in laparoscopic surgery. Our scenario is that the surgeon manipulates a sensor probe, makes a decision based on the feedback provided from the sensor, while simultaneously, the algorithm analyzes the time series of the sensor output. Thus, the algorithm assists the surgeon in making decisions by providing independent detection results. A deep neural network model with three hidden layers was used to analyze the sensor output. We propose methods to input the time series of the sensor output to the model for real-time analysis, and to determine the criterion for detection by the model. This study aims to validate the feasibility of the algorithm by using data acquired in our previous psychophysical experiment. There, novice participants were asked to detect a phantom of an early-stage gastric tumor through visual feedback from the tactile sensor. In addition to the analysis of the accuracy, signal detection theory was employed to assess the potential detection performance of the model. The detection performance was compared with that of human participants. We conducted two types of validation, and found that the detection performance of the model was not significantly different from that of the human participants if the data from a known user was included in the model construction. The result supports the feasibility of the proposed algorithm for detection assistance in computer-aided palpation.

Force estimation from OCT volumes using 3D CNNs

PURPOSE

Estimating the interaction forces of instruments and tissue is of interest, particularly to provide haptic feedback during robot-assisted minimally invasive interventions. Different approaches based on external and integrated force sensors have been proposed. These are hampered by friction, sensor size, and sterilizability. We investigate a novel approach to estimate the force vector directly from optical coherence tomography image volumes.

METHODS

We introduce a novel Siamese 3D CNN architecture. The network takes an undeformed reference volume and a deformed sample volume as an input and outputs the three components of the force vector. We employ a deep residual architecture with bottlenecks for increased efficiency. We compare the Siamese approach to methods using difference volumes and two-dimensional projections. Data were generated using a robotic setup to obtain ground-truth force vectors for silicon tissue phantoms as well as porcine tissue.

RESULTS

Our method achieves a mean average error of [Formula: see text] when estimating the force vector. Our novel Siamese 3D CNN architecture outperforms single-path methods that achieve a mean average error of [Formula: see text]. Moreover, the use of volume data leads to significantly higher performance compared to processing only surface information which achieves a mean average error of [Formula: see text]. Based on the tissue dataset, our methods shows good generalization in between different subjects.

CONCLUSIONS

We propose a novel image-based force estimation method using optical coherence tomography. We illustrate that capturing the deformation of subsurface structures substantially improves force estimation. Our approach can provide accurate force estimates in surgical setups when using intraoperative optical coherence tomography.

A Three Revolute-Revolute-Spherical Wearable Fingertip Cutaneous Device for Stiffness Rendering

We present a novel three Revolute-Revolute-Spherical (3RRS) wearable fingertip device for the rendering of stiffness information. It is composed of a static upper body and a mobile end-effector. The upper body is located on the nail side of the finger, supporting three small servo motors, and the mobile end-effector is in contact with the finger pulp. The two parts are connected by three articulated legs, actuated by the motors. The end-effector can move toward the user's fingertip and rotate it to simulate contacts with arbitrarily-oriented surfaces. Moreover, a vibrotactile motor placed below the end-effector conveys vibrations to the fingertip. The proposed device weights 25 g for 35 x 50 x 48 mm dimensions. To test the effectiveness of our wearable haptic device and its level of wearability, we carried out two experiments, enrolling 30 human subjects in total. The first experiment tested the capability of our device in differentiating stiffness information, while the second one focused on evaluating its applicability in an immersive virtual reality scenario. Results showed the effectiveness of the proposed wearable solution, with a JND for stiffness of 208.5   17.2 N/m. Moreover, all subjects preferred the virtual interaction experience when provided with wearable cutaneous feedback, even if results also showed that subjects found our device still a bit difficult to use.