Haptic feedback in the da Vinci Research Kit (dVRK): A user study based on grasping, palpation, and incision tasks

Investigating Haptic Feedback in Vision-Deficient Millirobot Telemanipulation

The evolution of magnetically actuated millirobots gives rise to unique teleoperation challenges due to their non-traditional kinematic and dynamic architectures, as well as their frequent use of suboptimal imaging modalities. Recent investigations into haptic interfaces for millirobots have shown promise but lack the clinically motivated task scenarios necessary to justify future development. In this work, we investigate the utility of haptic feedback on bilateral teleoperation of a magnetically actuated millirobot in visually deficient conditions. We conducted an N=23 user study in an aneurysm coiling inspired procedure, which required participants to navigate the robot through a maze in near total darkness to manipulate beads to a target under simulated fluoroscopy. We hypothesized that users will be better able to complete the telemanipulation task with haptic feedback while reducing excess forces on their surroundings compared to the no feedback conditions. Our results showed an over 40% improvement in participants' bead scoring, a nearly 10% reduction in mean force, and 13% reduction in maximum force with haptic feedback, as well as significant improvements in other metrics. Results highlight that benefits of haptic feedback are retained when haptic feedback is removed. These findings suggest that haptic feedback has the potential to significantly improve millirobot telemanipulation and control in traditionally vision deficient tasks.

Review of robotic systems for thoracoabdominal puncture interventional surgery

Cancer, with high morbidity and high mortality, is one of the major burdens threatening human health globally. Intervention procedures via percutaneous puncture have been widely used by physicians due to its minimally invasive surgical approach. However, traditional manual puncture intervention depends on personal experience and faces challenges in terms of precisely puncture, learning-curve, safety and efficacy. The development of puncture interventional surgery robotic (PISR) systems could alleviate the aforementioned problems to a certain extent. This paper attempts to review the current status and prospective of PISR systems for thoracic and abdominal application. In this review, the key technologies related to the robotics, including spatial registration, positioning navigation, puncture guidance feedback, respiratory motion compensation, and motion control, are discussed in detail.

Enhancing robotic telesurgery with sensorless haptic feedback

PURPOSE

This paper evaluates user performance in telesurgical tasks with the da Vinci Research Kit (dVRK), comparing unilateral teleoperation, bilateral teleoperation with force sensors and sensorless force estimation.

METHODS

A four-channel teleoperation system with disturbance observers and sensorless force estimation with learning-based dynamic compensation was developed. Palpation experiments were conducted with 12 users who tried to locate tumors hidden in tissue phantoms with their fingers or through handheld or teleoperated laparoscopic instruments with visual, force sensor, or sensorless force estimation feedback. In a peg transfer experiment with 10 users, the contribution of sensorless haptic feedback with/without learning-based dynamic compensation was assessed using NASA TLX surveys, measured free motion speeds and forces, environment interaction forces as well as experiment completion times.

RESULTS

The first study showed a 30% increase in accuracy in detecting tumors with sensorless haptic feedback over visual feedback with only a 5-10% drop in accuracy when compared with sensor feedback or direct instrument contact. The second study showed that sensorless feedback can help reduce interaction forces due to incidental contacts by about 3 times compared with unilateral teleoperation. The cost is an increase in free motion forces and physical effort. We show that it is possible to improve this with dynamic compensation.

CONCLUSION

We demonstrate the benefits of sensorless haptic feedback in teleoperated surgery systems, especially with dynamic compensation, and that it can improve surgical performance without hardware modifications.

A multimodal virtual vision platform as a next-generation vision system for a surgical robot

Robot-assisted surgery platforms are utilized globally thanks to their stereoscopic vision systems and enhanced functional assistance. However, the necessity of ergonomic improvement for their use by surgeons has been increased. In surgical robots, issues with chronic fatigue exist owing to the fixed posture of the conventional stereo viewer (SV) vision system. A head-mounted display was adopted to alleviate the inconvenience, and a virtual vision platform (VVP) is proposed in this study. The VVP can provide various critical data, including medical images, vital signs, and patient records, in three-dimensional virtual reality space so that users can access medical information simultaneously. An availability of the VVP was investigated based on various user evaluations by surgeons and novices, who executed the given tasks and answered questionnaires. The performances of the SV and VVP were not significantly different; however, the craniovertebral angle of the VVP was 16.35° higher on average than that of the SV. Survey results regarding the VVP were positive; participants indicated that the optimal number of displays was six, preferring the 2 × 3 array. Reflecting the tendencies, the VVP can be a neoconceptual candidate to be customized for medical use, which opens a new prospect in a next-generation surgical robot.

A Lightweight and Affordable Wearable Haptic Controller for Robot-Assisted Microsurgery

In robot-assisted microsurgery (RAMS), surgeons often face the challenge of operating with minimal feedback, particularly lacking in haptic feedback. However, most traditional desktop haptic devices have restricted operational areas and limited dexterity. This report describes a novel, lightweight, and low-budget wearable haptic controller for teleoperated microsurgical robotic systems. We designed a wearable haptic interface entirely made using off-the-shelf material-PolyJet Photopolymer, fabricated using liquid and solid hybrid 3D co-printing technology. This interface was designed to resemble human soft tissues and can be wrapped around the fingertips, offering direct contact feedback to the operator. We also demonstrated that the device can be easily integrated with our motion tracking system for remote microsurgery. Two motion tracking methods, marker-based and marker-less, were compared in trajectory-tracking experiments at different depths to find the most effective motion tracking method for our RAMS system. The results indicate that within the 4 to 8 cm tracking range, the marker-based method achieved exceptional detection rates. Furthermore, the performance of three fusion algorithms was compared to establish the unscented Kalman filter as the most accurate and reliable. The effectiveness of the wearable haptic controller was evaluated through user studies focusing on the usefulness of haptic feedback. The results revealed that haptic feedback significantly enhances depth perception for operators during teleoperated RAMS.

The benefits of haptic feedback in robot assisted surgery and their moderators: a meta-analysis

Robot assisted surgery (RAS) provides medical practitioners with valuable tools, decreasing strain during surgery and leading to better patient outcomes. While the loss of haptic sensation is a commonly cited disadvantage of RAS, new systems aim to address this problem by providing artificial haptic feedback. N = 56 papers that compared robotic surgery systems with and without haptic feedback were analyzed to quantify the performance benefits of restoring the haptic modality. Additionally, this study identifies factors moderating the effect of restoring haptic sensation. Overall results showed haptic feedback was effective in reducing average forces (Hedges' g = 0.83) and peak forces (Hedges' g = 0.69) applied during surgery, as well as reducing the completion time (Hedges' g = 0.83). Haptic feedback has also been found to lead to higher accuracy (Hedges' g = 1.50) and success rates (Hedges' g = 0.80) during surgical tasks. Effect sizes on several measures varied between tasks, the type of provided feedback, and the subjects' levels of surgical expertise, with higher levels of expertise generally associated with smaller effect sizes. No significant differences were found between virtual fixtures and rendering contact forces. Implications for future research are discussed.

Integration of Reinforcement Learning in a Virtual Robotic Surgical Simulation

Background. The revolutions in AI hold tremendous capacity to augment human achievements in surgery, but robust integration of deep learning algorithms with high-fidelity surgical simulation remains a challenge. We present a novel application of reinforcement learning (RL) for automating surgical maneuvers in a graphical simulation.Methods. In the Unity3D game engine, the Machine Learning-Agents package was integrated with the NVIDIA FleX particle simulator for developing autonomously behaving RL-trained scissors. Proximal Policy Optimization (PPO) was used to reward movements and desired behavior such as movement along desired trajectory and optimized cutting maneuvers along the deformable tissue-like object. Constant and proportional reward functions were tested, and TensorFlow analytics was used to informed hyperparameter tuning and evaluate performance.Results. RL-trained scissors reliably manipulated the rendered tissue that was simulated with soft-tissue properties. A desirable trajectory of the autonomously behaving scissors was achieved along 1 axis. Proportional rewards performed better compared to constant rewards. Cumulative reward and PPO metrics did not consistently improve across RL-trained scissors in the setting for movement across 2 axes (horizontal and depth).Conclusion. Game engines hold promising potential for the design and implementation of RL-based solutions to simulated surgical subtasks. Task completion was sufficiently achieved in one-dimensional movement in simulations with and without tissue-rendering. Further work is needed to optimize network architecture and parameter tuning for increasing complexity.

5G-Based Telerobotic Ultrasound System Improves Access to Breast Examination in Rural and Remote Areas: A Prospective and Two-Scenario Study

OBJECTIVE

Ultrasound (US) plays an important role in the diagnosis and management of breast diseases; however, effective breast US screening is lacking in rural and remote areas. To alleviate this issue, we prospectively evaluated the clinical availability of 5G-based telerobotic US technology for breast examinations in rural and remote areas.

METHODS

Between September 2020 and March 2021, 63 patients underwent conventional and telerobotic US examinations in a rural island (Scenario A), while 20 patients underwent telerobotic US examination in a mobile car located in a remote county (Scenario B) in May 2021. The safety, duration, US image quality, consistency, and acceptability of the 5G-based telerobotic US were assessed.

RESULTS

In Scenario A, the average duration of the telerobotic US procedure was longer than that of conventional US (10.3 ± 3.3 min vs. 7.6 ± 3.0 min, p = 0.017), but their average imaging scores were similar (4.86 vs. 4.90, p = 0.159). Two cases of gynecomastia, one of lactation mastitis, and one of postoperative breast effusion were diagnosed and 32 nodules were detected using the two US methods. There was good interobserver agreement between the US features and BI-RADS categories of the identical nodules (ICC = 0.795-1.000). In Scenario B, breast nodules were detected in 65% of the patients using telerobotic US. Its average duration was 10.1 ± 2.3 min, and the average imaging score was 4.85. Overall, 90.4% of the patients were willing to choose telerobotic US in the future, and tele-sonologists were satisfied with 85.5% of the examinations.

CONCLUSION

The 5G-based telerobotic US system is feasible for providing effective breast examinations in rural and remote areas.

Clinical application of a 5G-based telerobotic ultrasound system for thyroid examination on a rural island: a prospective study

PURPOSE

To evaluate the feasibility of a 5G-based telerobotic ultrasound (US) system for thyroid examination on a rural island.

METHODS

From September 2020 to March 2021, this prospectively study enrolled a total of 139 patients (average age, 58.6 ± 12.7 years) included 33 males and 106 females, who underwent 5G-based telerobotic thyroid US examination by a tele-doctor at Shanghai Tenth People's Hospital and a conventional thyroid US examination at Chongming Second People's Hospital 84 km away. The clinical feasibility of 5G-based telerobotic US for thyroid examination were evaluated in terms of safety, duration, US image quality, diagnostic results, and questionnaire survey.

RESULTS

92.8% of patients had no examination-related complaints. The average duration of the 5G-based telerobotic US examination was similar as that of conventional US examination (5.57 ± 2.20 min vs. 5.23 ± 2.1 min, P = 0.164). The image quality of telerobotic US correlated well with that of conventional US (4.63 ± 0.60 vs. 4.65 ± 0.61, P = 0.102). There was no significant difference between two types of US examination methods for the diameter measurement of the thyroid, cervical lymph nodes, and thyroid nodules. Two lymphadenopathies and 20 diffuse thyroid diseases were detected in two types of US methods. 124 thyroid nodules were detected by telerobotic US and 127 thyroid nodules were detected by conventional US. Among them, 122 were the same thyroid nodules. In addition, there were good consistency in the US features (component, echogenicity, shape, and calcification) and ACR TI-RADS category of the same thyroid nodules between telerobotic and conventional US examinations (ICC = 0.788-0.863). 85.6% of patients accepted the telerobotic US, and 87.1% were willing to pay extra fee for the telerobotic US.

CONCLUSION

The 5G-based telerobotic US system can be a routine diagnostic tool for thyroid examination for patients on a rural island.

Robot-assisted vs. video-assisted thoracoscopic surgery in lung cancer

Background

The major advantages of robot-assisted surgery are the fine field of view provided by the high-precision three-dimensional (3D) images and the good operability provided by the robotic arms that enables precise movements. A growing number of retrospective studies have compared robotic-assisted thoracoscopic surgery (RATS) with video-assisted thoracoscopic surgery (VATS), but the number of cases is limited and the results are contradictory.

Methods

We studied the medical records of primary lung cancer patients who underwent lobectomy with lymph node dissection between 2017 and 2020. Four hundred and eleven patients fulfilled the inclusion criteria in this study (RATS: 103; VATS: 308). We compared the perioperative factors and postoperative results of the VATS and RATS groups. Further, we adjusted background factors using propensity score matching (PSM) then compared the results of 200 patients (100 patients in each group). In this study, we matched interlobar fissure completeness, which affects operative difficulty and operative time; however, this has been superficially compared in previous studies.

Results

After PSM, a significant difference was observed in the intraoperative blood loss (RATS: 53.3 mL, VATS: 120.3 mL, P=0.04). The rates of surgical complications were comparable between the groups (10.0% vs. 13.0%, P=0.66) with similar mean operation times (RATS: 215.0 min, VATS: 210.1 min, P=0.57). The mean postoperative stay in the RATS group was shorter than that in the VATS group (10.0 vs. 11.5 days, P=0.04).

Conclusions

Initial experience of RATS had no obvious drawbacks when compared with that of VATS on propensity-matched analysis.

Position Control and Force Estimation Method for Surgical Forceps Using SMA Actuators and Sensors

Minimally invasive surgery is increasingly used in many medical operations because of the benefits for the patients. However, for the surgeons, accessing the situs through a small incision or natural orifice comes with a reduction of the degrees of freedom of the instrument. Due to friction of the mechanical coupling, the haptic feedback lacks sensitivity that could lead to damage of the tissue. The approach of this work to overcome these problems is to develop a control concept for position control and force estimation with shape memory alloys (SMA) which could offer haptic feedback in a novel handheld instrument. The concept aims to bridge the gap between manually actuated laparoscopic instruments and surgical robots. Nickel-titanium shape memory alloys are used for actuation because of their high specific energy density. The work includes the manufacturing of a functional model as a proof of concept comprising the development of a suitable forceps mechanism and electronic circuit for position control and gripping force measurement, as well as designing an ergonomic user interface with haptic force feedback.

Enhancing the Localization of Uterine Leiomyomas Through Cutaneous Softness Rendering for Robot-Assisted Surgical Palpation Applications

Integrating tactile feedback for lump localization in Robot-assisted Minimally Invasive Surgery (RMIS) represents an open research issue, which is still far to be solved. Main reasons for this are related e.g. to the need for a transparent connection with the teleoperating console, and an intuitive decoding of the delivered information. In this article, we focus on the specific case of RMIS treatment of uterine leiomyomas or fibroids, where little has been done in haptics to improve the outcomes of robotics-enabled palpation tasks. In this article, we propose the usage of a wearable haptic interface for softness rendering as a lump display. The device was integrated in a teleoperation architecture that simulates a robot-assisted surgical palpation task of leiomyomas. This article moved from an ex-vivo sample characterization of uterine tissues to show the effectiveness of our interface in conveying meaningful softness information. We extensively tested our system with gynecologic surgeons in palpation tasks with silicone specimens, which replicated the characteristics of uterine tissues with embedded leyomiomas. Results show that our system enables a softness-based discrimination of the embedded fibroids comparable to the one that physicians would achieve using directly their fingers in palpation tasks. Furthermore, the feedback provided by the haptic interface was perceived as comfortable, intuitive, and highly useful for fibroid localization.

Bio-inspired Haptic Feedback for Artificial Palpation in Robotic Surgery

Adding haptic feedback has been reported to improve the outcome of minimally invasive robotic surgery. In this study, we seek to determine whether an algorithm based on simulating responses of a cutaneous afferent population can be implemented to improve the performance of presenting haptic feedback for robot-assisted surgery. We propose a bio-inspired controlling model to present vibration and force feedback to help surgeons localize underlying structures in phantom tissue. A single pair of actuators was controlled by outputs of a model of a population of cutaneous afferents based on the pressure signal from a single sensor embedded in surgical forceps. We recruited 25 subjects including 10 expert surgeons to evaluate the performance of the bio-inspired controlling model in an artificial palpation task using the da Vinci surgical robot. Among the control methods tested, the bio-inspired system was unique in allowing both novices and experts to easily identify the locations of all classes of tumors and did so with reduced contact force and tumor contact time. This work demonstrates the utility of our bio-inspired multi-modal feedback system, which resulted in superior performance for both novice and professional users, in comparison to a traditional linear and the existing piecewise discrete algorithms of haptic feedback.

An Application-Based Review of Haptics Technology

Recent technological development has led to the invention of different designs of haptic devices, electromechanical devices that mediate communication between the user and the computer and allow users to manipulate objects in a virtual environment while receiving tactile feedback. The main criteria behind providing an interactive interface are to generate kinesthetic feedback and relay information actively from the haptic device. Sensors and feedback control apparatus are of paramount importance in designing and manufacturing a haptic device. In general, haptic technology can be implemented in different applications such as gaming, teleoperation, medical surgeries, augmented reality (AR), and virtual reality (VR) devices. This paper classifies the application of haptic devices based on the construction and functionality in various fields, followed by addressing major limitations related to haptics technology and discussing prospects of this technology.

Robotic surgical systems in urology: What is currently available?

Robotic assistance in laparoscopic surgery was introduced at the turn of this millennium, marking a milestone in the history of surgery. Urologists were early adopters of robotic technology and the indications of robot-assisted surgery in urology are expanding. Over the last 20 years, the da Vinci surgical system was the dominant system in the robotic surgical market. However, the recent expiration of Intuitive patents has allowed new systems to enter the market more freely. We performed a nonsystematic literature review using the PubMed/MEDLINE search engines. The aim of this review was to briefly summarize the currently available robotic surgical systems for laparoscopic urologic surgery. New surgical devices have already been launched in the robotic market and the da Vinci systems have some competition. The innovation of robotic technology is continuing, and new features such as an open-console design, haptic feedback, smaller instruments, and separately mounted robotic arms have been introduced. A new robotic era is rising, and new systems and technologies enhancing patient care are welcomed.

Haptic Intracorporeal Palpation Using a Cable-Driven Parallel Robot: A User Study

OBJECTIVE

Intraoperative palpation is a surgical gesture jeopardized by the lack of haptic feedback which affects robotic minimally invasive surgery. Restoring the force reflection in teleoperated systems may improve both surgeons' performance and procedures' outcome.

METHODS

A force-based sensing approach was developed, based on a cable-driven parallel manipulator with anticipated seamless and low-cost integration capabilities in teleoperated robotic surgery. No force sensor on the end-effector is used, but tissue probing forces are estimated from measured cable tensions. A user study involving surgical trainees (n = 22) was conducted to experimentally evaluate the platform in two palpation-based test-cases on silicone phantoms. Two modalities were compared: visual feedback alone and both visual + haptic feedbacks available at the master site.

RESULTS

Surgical trainees' preference for the modality providing both visual and haptic feedback is corroborated by both quantitative and qualitative metrics. Hard nodules detection sensitivity improves (94.35 ± 9.1% vs 76.09 ± 19.15% for visual feedback alone), while also exerting smaller forces (4.13 ± 1.02 N vs 4.82 ± 0.81 N for visual feedback alone) on the phantom tissues. At the same time, the subjective perceived workload decreases.

CONCLUSION

Tissue-probe contact forces are estimated in a low cost and unique way, without the need of force sensors on the end-effector. Haptics demonstrated an improvement in the tumor detection rate, a reduction of the probing forces, and a decrease in the perceived workload for the trainees.

SIGNIFICANCE

Relevant benefits are demonstrated from the usage of combined cable-driven parallel manipulators and haptics during robotic minimally invasive procedures. The translation of robotic intraoperative palpation to clinical practice could improve the detection and dissection of cancer nodules.

A review of haptic feedback in tele-operated robotic surgery

During traditional surgery, the surgeons' hands are in direct contact with organs, and surgeons rely on the sense of touch to perform surgery. In teleoperated robotic systems, all physical connections between the surgeon and both the robot and patient, are absent. The surgeon must estimate the force exerted on organs, based only on visual deformation of tissues he is pulling, pushing, gripping, or suturing. It is hard to imagine how to operate with no haptic sensations, and it is surprising that commercially available robots didn't include until now any Haptic Feedback, despite reports about tissue injury, and inability to perform complex manipulation. The sense of touch must be created by stimuli sensed by the surgeon. Haptic sensors are required to collect and send haptic information, and display them on the operator's side, creating telepresence, known as transparency. Multiple ways have been developed to improve transparency through force feedback and tactile feedback. However, this interferes with the stability of the closed-loop controlling interactions between master, robot and remote environment. Cutaneous feedback is more stable and less transparent; force feedback is more transparent and less stable. Thus, multimodal platforms of haptic feedback would try to find the best trade-off between both modalities.