Effects of visual force feedback on robot-assisted surgical task performance

Vision-based estimation of manipulation forces by deep learning of laparoscopic surgical images obtained in a porcine excised kidney experiment

In robot-assisted surgery, in which haptics should be absent, surgeons experience haptics-like sensations as "pseudo-haptic feedback". As surgeons who routinely perform robot-assisted laparoscopic surgery, we wondered if we could make these "pseudo-haptics" explicit to surgeons. Therefore, we created a simulation model that estimates manipulation forces using only visual images in surgery. This study aimed to achieve vision-based estimations of the magnitude of forces during forceps manipulation of organs. We also attempted to detect over-force, exceeding the threshold of safe manipulation. We created a sensor forceps that can detect precise pressure at the tips with three vectors. Using an endoscopic system that is used in actual surgery, images of the manipulation of excised pig kidneys were recorded with synchronized force data. A force estimation model was then created using deep learning. Effective detection of over-force was achieved if the region of the visual images was restricted by the region of interest around the tips of the forceps. In this paper, we emphasize the importance of limiting the region of interest in vision-based force estimation tasks.

Robotic mitral valve surgery

Totally endoscopic robotic mitral valve repair is the least invasive surgical therapy for mitral valve disease. Robotic mitral valve surgery demonstrates faster recovery with shorter hospital stays, less morbidity, and equivalent mortality and mid-term durability compared to sternotomy. In this review, we will explore the advantages and disadvantages of robotic mitral valve surgery and consider important technical details of both operative set-up and mitral valve repair techniques. The number of robotic cardiac surgical procedures being performed globally is expected to continue to rise as experience grows with robotic techniques and increasing numbers of cardiac surgeons become proficient with this innovative technology. This will be facilitated by the introduction of newer robotic systems and increasing patient demand.

IDEAL-D Phase 0 Evaluation of the Avatera System in Robot-Assisted Prostate, Bladder and Renal Surgery

Purpose: To evaluate the utilization of novel Avatera system in urological operations according to the IDEAL-D framework recommendations for high-risk invasive surgical devices. Materials and Methods: Three surgeons attempted to perform 23 upper and lower urinary tract operations on human cadavers and in live porcine models using the Avatera system. Total operative time and the duration of the substeps were evaluated. Surgical performance was assessed with the Global Evaluative Assessment of Robotic Skills (GEARS) score. Suturing was rated using the technical checklist for the assessment of suturing in robotic surgery. Attending surgeons rated their satisfaction with the Avatera system on a scale of 1-5. Results and Limitation: Seventeen out of 18 operations performed on cadavers were completed, while one pyeloplasty was discontinued. All five operations performed in porcine models were completed. Although 1 pig was euthanized on the fifth postoperative day, its symptoms were unrelated to surgery. Mean GEARS and Suturing scores in the upper urinary tract were 29 ± 0.7 and 29.5 ± 0.95, respectively, and in the lower urinary 28.5 ± 1.2 and 29.5 ± 0.5, respectively. Surgeons' satisfaction was high or very high for all procedures. Conclusions: The Avatera system was associated with good surgical performance and high surgeons' satisfaction rates. All urological procedures performed were shown to be feasible, with comparable risks to other robot-assisted surgery systems.

Effects of a force feedback function in a surgical robot on the suturing procedure

BACKGROUND

Currently, widely used robotic surgical systems do not provide force feedback. This study aimed to evaluate the impact and benefits of a force feedback function on the suturing procedure.

METHODS

Twenty surgeons were recruited and divided into young (Y-group, n = 11) and senior (S-group, n = 9) groups, based on their years of surgical experience. The effect of the force feedback function on suturing quality was evaluated using an objective assessment system (A-LAP mini, Kyoto Kagaku Co., Ltd., Kyoto, Japan). Each participant completed the suturing task on intestinal model sheets with the robotic contact force feedback on and off. The task accomplishment time (s), maximal force (Newton, N) applied to the robotic forceps, and quality of suturing (assessed by A-LAP mini) were recorded as performance parameters.

RESULTS

In total, the maximal force applied to the robotic forceps was significantly decreased with the robotic force feedback switched on (median [interquartile range]: 2.8 N (2.3-3.2)) as compared with when the feedback was switched off (3.4 N (2.7-4.0), P < 0.001). The contact force feedback function did not affect the objectively assessed suturing score (18 points (17.7-19.0) versus 18 points (17.0-19.0), P = 0.421). The contact force feedback function slightly shortened the task accomplishment time in the Y-group (552.5 s (466.5-832) versus 605.5 s (476.2-689.7), P = 0.851) but not in the S-group (566 s (440.2-703.5) versus 470.5 s (419.7-560.2), P = 0.164).

CONCLUSIONS

With the contact force feedback function, the suturing task was completed with a smaller maximal force, while maintaining the quality of suturing. Because the benefits are more apparent in young surgeons, robots with the contact force feedback function will facilitate the educational process in novice surgeons.

Enhanced grip force estimation in robotic surgery: A sparrow search algorithm-optimized backpropagation neural network approach

The absence of an effective gripping force feedback mechanism in minimally invasive surgical robot systems impedes physicians' ability to accurately perceive the force between surgical instruments and human tissues during surgery, thereby increasing surgical risks. To address the challenge of integrating force sensors on minimally invasive surgical tools in existing systems, a clamping force prediction method based on mechanical clamp blade motion parameters is proposed. The interrelation between clamping force, displacement, compression speed, and the contact area of the clamp blade indenter was analyzed through compression experiments conducted on isolated pig kidney tissue. Subsequently, a prediction model was developed using a backpropagation (BP) neural network optimized by the Sparrow Search Algorithm (SSA). This model enables real-time prediction of clamping force, facilitating more accurate estimation of forces between instruments and tissues during surgery. The results indicate that the SSA-optimized model outperforms traditional BP networks and genetic algorithm-optimized (GA) BP models in terms of both accuracy and convergence speed. This study not only provides technical support for enhancing surgical safety and efficiency, but also offers a novel research direction for the design of force feedback systems in minimally invasive surgical robots in the future.

Impact of a pneumatic surgical robot with haptic feedback function on surgical manipulation

Although robotic-assisted surgery has the advantages of low patient burden and high precision without unsteady hand movements, the lack of tactile sensations may result in unexpected iatrogenic organ damage. The Saroa (Riverfield Inc., Tokyo, Japan) is a pneumatically driven robot that provides real-time haptic feedback to the surgeon. Using the Saroa robot, six examinees performed puffed rice transfer and four of them performed pig lung resection tasks with the feedback function turned on and off. The puffed rice transfer task consisted of transferring 20 grains of puffed rice from the left to the right compartment in the training box. The mean grasping forces during the puffed rice transfer task with the haptic feedback function turned off and on were 2.14 N and 0.63 N, respectively (P = 0.003). The mean grasping forces during the pig lung resection task were lower with the feedback turned on than turned off. The force that the forceps exerted on the grasping object was weaker in both tasks when the haptic feedback function was turned on, suggesting that the feedback function allows gentler handling of tissues, improving patient safety during robotic surgery.

Sensory substitution increases robotic surgical performance and sets the ground for a mediating role of the sense of embodiment: a systematic review

Sensory Substitution (SS) allows the elaboration of information via non preferential sensory modalities. This phenomenon occurs in robotic-assisted surgery (RAS), in which haptic feedback is lacking. It has been suggested that SS could sustain surgeons' proficiency by means of visual clues for inferring tactile information, that also promotes the feeling of haptic phantom sensations. A critical role in reaching a good performance in procedural tasks is also sustained by the Sense of Embodiment (SE), that is, the capacity to integrate objects into subjective bodily self-representation. As SE is enhanced by haptic sensations, we hypothesize a role of SS in promoting SE in RAS. Accordingly, the goal of this systematic review is to summarize the evidence pertaining the study of SS in RAS in order to highlight the impact on the performance, and to identify a mediating role of the SE in increasing dexterity in RAS. Eight studies selected from the MEDLINE and Scopus® databases met inclusion criteria for a qualitative synthesis. Results indicated that haptic to other modalities SS enhanced force consistency and accuracy, and decreased surgeon fatigue. Expert surgeons, as compared to novices, showed a better natural SS processing, testified by a proficient performance with and without SS aids. No studies investigated the mediating role of SE. These findings indicate that SS is subjected to learning and memory processes that help surgeons to rapidly derive haptic-correlates from visual clues, which are highly required for a good performance. Also, the higher ability of doing SS and the associated perception of haptic sensations might increase multisensory integration, which might sustain performance.

Three Dimensional Printing Technology Used to Create a High-Fidelity Ureteroscopy Simulator: Development and Validity Assessment (Rein-3D-Print-UroCCR-39)

OBJECTIVE

To create and assess the validity of a high-fidelity, three dimensional (3D) printed, flexible ureteroscopy simulator resulting from a real case.

METHODS

A patient's CT scan was segmented to obtain a 3D model in .stl format, including the urinary bladder, ureter and renal cavities. The file was printed and a kidney stone was introduced into the cavities. The simulated surgery consisted of monobloc stone extraction. Nineteen participants split into 3 groups according to their level (6 medical students, 7 residents and 6 urology fellows) performed the procedure twice at a 1-month interval. They were rated according to a global score and a task-specific score, based on an anonymized, timed video recording.

RESULTS

Participants demonstrated a significant improvement between the 2 assessments, both on the global score (29.4 vs 21.9 points out of 35; P < .001) and the task-specific score (17.7 vs 14.7 points out of 20; P < .001) as well as procedure time (498.5 vs 700 seconds; P = .001). Medical students showed the greatest progress for the global score (+15.5 points (mean), P = .001) and the task-specific score (+6.5 points (mean), P < .001). 69.2% of participants considered the model as visually quite realistic or highly realistic and all of them judged it quite or extremely interesting for intern training purposes.

CONCLUSION

Our 3D printed ureteroscopy simulator was able to enhance the progress of medical students who are new to endoscopy, whilst being valid and reasonably priced. It could become part of a training program in urology, in line with the latest recommendations for surgical education.

Effects of automated skill assessment on robotic surgery training

BACKGROUND

Several automated skill-assessment approaches have been proposed for robotic surgery, but their utility is not well understood. This article investigates the effects of one machine-learning-based skill-assessment approach on psychomotor skill development in robotic surgery training.

METHODS

N = 29 trainees (medical students and residents) with no robotic surgery experience performed five trials of inanimate peg transfer with an Intuitive Surgical da Vinci Standard robot. Half of the participants received no post-trial feedback. The other half received automatically calculated scores from five Global Evaluative Assessment of Robotic Skill domains post-trial.

RESULTS

There were no significant differences between the groups regarding overall improvement or skill improvement rate. However, participants who received post-trial feedback rated their overall performance improvement significantly lower than participants who did not receive feedback.

CONCLUSIONS

These findings indicate that automated skill evaluation systems might improve trainee self-awareness but not accelerate early stage psychomotor skill development in robotic surgery training.

Inducing Performance of Commercial Surgical Robots in Space

Pre-existing surgical robotic systems are sold with electronics (sensors and controllers) that can prove difficult to retroactively improve when newly developed methods are proposed. Improvements must be somehow "imposed" upon the original robotic systems. What options are available for imposing performance from pre-existing, common systems and how do the options compare? Optimization often assumes idealized systems leading to open-loop results (lacking feedback from sensors), and this manuscript investigates utility of prefiltering, such other modern methods applied to non-idealized systems, including fusion of noisy sensors and so-called "fictional forces" associated with measurement of displacements in rotating reference frames. A dozen modern approaches are compared as the main contribution of this work. Four methods are idealized cases establishing a valid theoretical comparative benchmark. Subsequently, eight modern methods are compared against the theoretical benchmark and against the pre-existing robotic systems. The two best performing methods included one modern application of a classical approach (velocity control) and one modern approach derived using Pontryagin's methods of systems theory, including Hamiltonian minimization, adjoint equations, and terminal transversality of the endpoint Lagrangian. The key novelty presented is the best performing method called prefiltered open-loop optimal + transport decoupling, achieving 1-3 percent attitude tracking performance of the robotic instrument with a two percent reduced computational burden and without increased costs (effort).

Feasibility of telesurgery in the modern era

Telesurgery is not a foreign concept and dates to as early as the 1920s. The use of robots in medicine has had a very positive effect and improved outcomes with little to no adverse effects. Having global access to telemedicine and telesurgery during the COVID-19 pandemic and being able to provide top medical care to gravely ill and contagious patients without compromising the safety of the medical team would be a very big achievement. We explore the hurdles needed to make it a realistic goal and give recommendations to achieve it utilizing the major advancements that have occurred over the past few years in the fields of engineering, communication etc. The biggest issues needed to be addressed are of financial investment, legal concerns, and availability of high-speed uninterrupted data connections.

Visual Haptic Feedback for Training of Robotic Suturing

Current surgical robotic systems are teleoperated and do not have force feedback. Considerable practice is required to learn how to use visual input such as tissue deformation upon contact as a substitute for tactile sense. Thus, unnecessarily high forces are observed in novices, prior to specific robotic training, and visual force feedback studies demonstrated reduction of applied forces. Simulation exercises with realistic suturing tasks can provide training outside the operating room. This paper presents contributions to realistic interactive suture simulation for training of suturing and knot-tying tasks commonly used in robotically-assisted surgery. To improve the realism of the simulation, we developed a global coordinate wire model with a new constraint development for the elongation. We demonstrated that a continuous modeling of the contacts avoids instabilities during knot tightening. Visual cues are additionally provided, based on the computation of mechanical forces or constraints, to support learning how to dose the forces. The results are integrated into a powerful system-agnostic simulator, and the comparison with equivalent tasks performed with the da Vinci Xi system confirms its realism.

Reducing retraction forces with tactile feedback during robotic total mesorectal excision in a porcine model

Excessive tissue-instrument interaction forces during robotic surgery have the potential for causing iatrogenic tissue damages. The current in vivo study seeks to assess whether tactile feedback could reduce intraoperative tissue-instrument interaction forces during robotic-assisted total mesorectal excision. Five subjects, including three experts and two novices, used the da Vinci robot to perform total mesorectum excision in four pigs. The grip force in the left arm, used for retraction, and the pushing force in the right arm, used for blunt pelvic dissection around the rectum, were recorded. Tissue-instrument interaction forces were compared between trials done with and without tactile feedback. The mean force exerted on the tissue was consistently higher in the retracting arm than the dissecting arm (3.72 ± 1.19 vs 0.32 ± 0.36 N, p < 0.01). Tactile feedback brought about significant reductions in average retraction forces (3.69 ± 1.08 N vs 4.16 ± 1.12 N, p = 0.02), but dissection forces appeared unaffected (0.43 ± 0.42 vs 0.37 ± 0.28 N, p = 0.71). No significant differences were found between retraction and dissection forces exerted by novice and expert robotic surgeons. This in vivo animal study demonstrated the efficacy of tactile feedback in reducing retraction forces during total mesorectal excision. Further research is required to quantify the clinical impact of such force reduction.

Position Control and Force Estimation Method for Surgical Forceps Using SMA Actuators and Sensors

Minimally invasive surgery is increasingly used in many medical operations because of the benefits for the patients. However, for the surgeons, accessing the situs through a small incision or natural orifice comes with a reduction of the degrees of freedom of the instrument. Due to friction of the mechanical coupling, the haptic feedback lacks sensitivity that could lead to damage of the tissue. The approach of this work to overcome these problems is to develop a control concept for position control and force estimation with shape memory alloys (SMA) which could offer haptic feedback in a novel handheld instrument. The concept aims to bridge the gap between manually actuated laparoscopic instruments and surgical robots. Nickel-titanium shape memory alloys are used for actuation because of their high specific energy density. The work includes the manufacturing of a functional model as a proof of concept comprising the development of a suitable forceps mechanism and electronic circuit for position control and gripping force measurement, as well as designing an ergonomic user interface with haptic force feedback.

Robotic mitral valve surgery: a review and tips for safely negotiating the learning curve

Totally endoscopic robotic mitral valve repair represents the least invasive surgical therapy for mitral valve disease. Comparative results for robotic mitral valve surgery against sternotomy are impressive, repeatedly demonstrating shorter hospital stay, faster return to normal activities, less morbidity and equivalent mortality and mid-term durability. We lack data comparing robotic approaches to totally endoscopic minimally invasive mitral valve surgery using 3D vision platforms. In this review, we explore the advantages and disadvantages of robotic mitral valve surgery and share technical tips that we have learned to help teams embarking on their robotic journey. We consider factors necessary for the successful implementation of a robotic programme including the importance of training a dedicated team, with the common goal to avoid any compromise in either patient safety or repair quality during the learning curve. As experience grows with robotic techniques and more cardiac surgeons become proficient with this innovative technology, the volume of robotic cardiac procedures around the world will increase helped by the introduction of new robotic systems and patient demand. Well informed patients will increasingly seek out the opportunity of robotic valve reconstruction in reference centres in the hands of a few highly experienced robotic surgeons.

Proposal and Evaluation of Visual Haptics for Manipulation of Remote Machine System

Remote machine systems have drawn a lot of attention owing to accelerations of virtual reality (VR), augmented reality (AR), and the fifth generation (5G) networks. Despite recent trends of developing autonomous systems, the realization of sophisticated dexterous hand that can fully replace human hands is considered to be decades away. It is also extremely difficult to reproduce the sensilla of complex human hands. On the other hand, it is known that humans can perceive haptic information from visual information even without any physical feedback as cross modal sensation between visual and haptics sensations or pseudo haptics. In this paper, we propose a visual haptic technology, where haptic information is visualized in more perceptual images overlaid at the contact points of a remote machine hand. The usability of the proposed visual haptics was evaluated by subject's brain waves aiming to find out a new approach for quantifying "sense of oneness." In our proof-of-concept experiments using VR, subjects are asked to operate a virtual arm and hand presented in the VR space, and the performance of the operation with and without visual haptics information as measured with brain wave sensing. Consequently, three results were verified. Firstly, the information flow in the brain were significantly reduced with the proposed visual haptics for the whole α, β, and θ-waves by 45% across nine subjects. This result suggests that superimposing visual effects may be able to reduce the cognitive burden on the operator during the manipulation for the remote machine system. Secondly, high correlation (Pearson correlation factor of 0.795 at a p-value of 0.011) was verified between the subjective usability points and the brainwave measurement results. Finally, the number of the task successes across sessions were improved in the presence of overlaid visual stimulus. It implies that the visual haptics image could also facilitate operators' pre-training to get skillful at manipulating the remote machine interface more quickly.

Review of Advanced Medical Telerobots

The advent of telerobotic systems has revolutionized various aspects of the industry and human life. This technology is designed to augment human sensorimotor capabilities to extend them beyond natural competence. Classic examples are space and underwater applications when distance and access are the two major physical barriers to be combated with this technology. In modern examples, telerobotic systems have been used in several clinical applications, including teleoperated surgery and telerehabilitation. In this regard, there has been a significant amount of research and development due to the major benefits in terms of medical outcomes. Recently telerobotic systems are combined with advanced artificial intelligence modules to better share the agency with the operator and open new doors of medical automation. In this review paper, we have provided a comprehensive analysis of the literature considering various topologies of telerobotic systems in the medical domain while shedding light on different levels of autonomy for this technology, starting from direct control, going up to command-tracking autonomous telerobots. Existing challenges, including instrumentation, transparency, autonomy, stochastic communication delays, and stability, in addition to the current direction of research related to benefit in telemedicine and medical automation, and future vision of this technology, are discussed in this review paper.

Vision-Based Suture Tensile Force Estimation in Robotic Surgery

Compared to laparoscopy, robotics-assisted minimally invasive surgery has the problem of an absence of force feedback, which is important to prevent a breakage of the suture. To overcome this problem, surgeons infer the suture force from their proprioception and 2D image by comparing them to the training experience. Based on this idea, a deep-learning-based method using a single image and robot position to estimate the tensile force of the sutures without a force sensor is proposed. A neural network structure with a modified Inception Resnet-V2 and Long Short Term Memory (LSTM) networks is used to estimate the suture pulling force. The feasibility of proposed network is verified using the generated DB, recording the interaction under the condition of two different artificial skins and two different situations (in vivo and in vitro) at 13 viewing angles of the images by changing the tool positions collected from the master-slave robotic system. From the evaluation conducted to show the feasibility of the interaction force estimation, the proposed learning models successfully estimated the tensile force at 10 unseen viewing angles during training.

Tactile Perception Technologies and Their Applications in Minimally Invasive Surgery: A Review

Minimally invasive surgery (MIS) has been the preferred surgery approach owing to its advantages over conventional open surgery. As a major limitation, the lack of tactile perception impairs the ability of surgeons in tissue distinction and maneuvers. Many studies have been reported on industrial robots to perceive various tactile information. However, only force data are widely used to restore part of the surgeon’s sense of touch in MIS. In recent years, inspired by image classification technologies in computer vision, tactile data are represented as images, where a tactile element is treated as an image pixel. Processing raw data or features extracted from tactile images with artificial intelligence (AI) methods, including clustering, support vector machine (SVM), and deep learning, has been proven as effective methods in industrial robotic tactile perception tasks. This holds great promise for utilizing more tactile information in MIS. This review aims to provide potential tactile perception methods for MIS by reviewing literatures on tactile sensing in MIS and literatures on industrial robotic tactile perception technologies, especially AI methods on tactile images.

Generalization of spatio-temporal deep learning for vision-based force estimation

Robot-assisted minimally-invasive surgery is increasingly used in clinical practice. Force feedback offers potential to develop haptic feedback for surgery systems. Forces can be estimated in a vision-based way by capturing deformation observed in 2D-image sequences with deep learning models. Variations in tissue appearance and mechanical properties likely influence force estimation methods’ generalization. In this work, we study the generalization capabilities of different spatial and spatio-temporal deep learning methods across different tissue samples. We acquire several data-sets using a clinical laparoscope and use both purely spatial and also spatio-temporal deep learning models. The results of this work show that generalization across different tissues is challenging. Nevertheless, we demonstrate that using spatio-temporal data instead of individual frames is valuable for force estimation. In particular, processing spatial and temporal data separately by a combination of a ResNet and GRU architecture shows promising results with a mean absolute error of 15.450 compared to 19.744 mN of a purely spatial CNN.

Impact of haptic feedback on applied intracorporeal forces using a novel surgical robotic system-a randomized cross-over study with novices in an experimental setup

Background

Most currently used surgical robots have no force feedback; the next generation displays forces visually. A novel single-port robotic surgical system called FLEXMIN has been developed. Through an outer diameter of 38 mm, two instruments are teleoperated from a surgeon’s control console including true haptic force feedback. One additional channel incorporates a telescope, another is free for special instrument functions.

Methods

This randomized cross-over study analyzed the effect of haptic feedback on the application of intracorporeal forces. In a standardized experiment setup, the subjects had to draw circles with the surgical robot as gently as possible. The applied forces, the required time spans, and predefined error rates were measured.

Results

Without haptic feedback, the maximum forces (median/IQR) were 6.43 N/2.96 N. With haptic feedback, the maximum forces were lower (3.57 N/1.94 N, p < 0.001). Also, the arithmetic means of the force progression (p < 0.001) and their standard deviations (p < 0.001) were lower. Not significant were the shorter durations and lower error rates. No sequence effect of force or duration was detected. No characteristic learning or fatigue curve was observed.

Conclusions

In the experiment setup, the true haptic force feedback can reduce the applied intracorporeal robotic force to one-half when considering the aspects maximum, means, and standard deviation. Other test tasks are needed to validate the influence of force feedback on surgical efficiency and safety.

Cadaveric feasibility study of a teleoperated parallel continuum robot with variable stiffness for transoral surgery

Robot-assisted technologies are overcoming the limitations of the current approaches for transoral surgeries, which are suffering from limited vision and workspace. As a result, we develop a novel teleoperated parallel continuum robot with variable stiffness for collision avoidance. This paper focuses on the feasibility study on a cadaveric model for the robotic system as a first trial. We introduce the configuration of the robotic system, the description of the processes of the trial, including the setting of the robotic system, the test of stiffness, and the action of the manipulation. The contact force between the manipulators with different stiffness and the surrounding tissues and a series of surgical operations of the manipulator, including grasping, cutting, pushing, and pulling tissues under the master-slave control mode, were recorded and analyzed. Experimental results suggest that the typical surgical procedure on a cadaveric model was successfully performed. Moreover, the efficacy and feasibility of the developed robotic system are verified to satisfy the requirements of transoral robotic surgery (TORS). Graphical abstract.

A review of haptic feedback in tele-operated robotic surgery

During traditional surgery, the surgeons' hands are in direct contact with organs, and surgeons rely on the sense of touch to perform surgery. In teleoperated robotic systems, all physical connections between the surgeon and both the robot and patient, are absent. The surgeon must estimate the force exerted on organs, based only on visual deformation of tissues he is pulling, pushing, gripping, or suturing. It is hard to imagine how to operate with no haptic sensations, and it is surprising that commercially available robots didn't include until now any Haptic Feedback, despite reports about tissue injury, and inability to perform complex manipulation. The sense of touch must be created by stimuli sensed by the surgeon. Haptic sensors are required to collect and send haptic information, and display them on the operator's side, creating telepresence, known as transparency. Multiple ways have been developed to improve transparency through force feedback and tactile feedback. However, this interferes with the stability of the closed-loop controlling interactions between master, robot and remote environment. Cutaneous feedback is more stable and less transparent; force feedback is more transparent and less stable. Thus, multimodal platforms of haptic feedback would try to find the best trade-off between both modalities.

Laparoscopic Robotic Surgery: Current Perspective and Future Directions

Just as laparoscopic surgery provided a giant leap in safety and recovery for patients over open surgery methods, robotic-assisted surgery (RAS) is doing the same to laparoscopic surgery. The first laparoscopic-RAS systems to be commercialized were the Intuitive Surgical, Inc. (Sunnyvale, CA, USA) da Vinci and the Computer Motion Zeus. These systems were similar in many aspects, which led to a patent dispute between the two companies. Before the dispute was settled in court, Intuitive Surgical bought Computer Motion, and thus owned critical patents for laparoscopic-RAS. Recently, the patents held by Intuitive Surgical have begun to expire, leading to many new laparoscopic-RAS systems being developed and entering the market. In this study, we review the newly commercialized and prototype laparoscopic-RAS systems. We compare the features of the imaging and display technology, surgeons console and patient cart of the reviewed RAS systems. We also briefly discuss the future directions of laparoscopic-RAS surgery. With new laparoscopic-RAS systems now commercially available we should see RAS being adopted more widely in surgical interventions and costs of procedures using RAS to decrease in the near future.

Electrically-Evoked Proximity Sensation Can Enhance Fine Finger Control in Telerobotic Pinch

For teleoperation tasks requiring high control accuracy, it is essential to provide teleoperators with information on the interaction between the end effector and the remote environment. Real-time imaging devices have been widely adopted, but it delivers limited information, especially when the end effectors approach the target following the line-of-sight. In such situations, teleoperators rely on the perspective at the screen and can apply high force unintentionally at the initial contact. This research proposes to deliver the distance information at teleoperation to the fingertips of teleoperators, i.e., proximity sensation. Transcutaneous electrical stimulation was applied onto the fingertips of teleoperators, with the pulsing frequency inversely proportional to the distance. The efficacy of the proximity sensation was evaluated by the initial contact force during telerobotic pinch in three sensory conditions: vision only, vision + visual assistance (distance on the screen), and vision + proximity sensation. The experiments were repeated at two viewing angles: 30–60° and line-of-sight, for eleven healthy human subjects. For both cases, the initial contact force could be significantly reduced by either visual assistance (20–30%) or the proximity sensation (60–70%), without additional processing time. The proximity sensation is two-to-three times more effective than visual assistance regarding the amount of force reduction.

Performance Evaluation of Optically Sensorized Tendons for Articulate Surgical Instruments

This study proposes an optically sensorized force-sensing tendon for minimally invasive surgical instruments. The tendon is composed of a high strength, polarization maintaining (PM) optical fiber with Bragg sensors (FBGs) that negate the cross-sensitivity of conventional FBGs. The PM-FBG fiber is locally reinforced with high stiffness Kevlar that enhances its load carrying capacity while enabling higher curvatures in tendon routing. The composite tendon has a mean diameter of ∼268 μm which preserves the form-factor of instruments within this scope. Importantly, the tendons can improve the functionality of such tools by enabling local force and tissue-resistance estimation. This paper explores the performance of these sensorized tendons in terms of strength, stability, response under dynamic load, friction, and sensitivity as a force measuring tool within an 18 Ga articulate Nitinol (NiTi) cannula (a proxy for potential applications). Results reaffirm the potential of a bi-modal sensing and actuation component within instruments for robotic surgery.

Bone Conduction Headphones for Force Feedback in Robotic Surgery

Bone conduction headphones (Fig. 1) offer the unique ability to provide auditory information to the user without obstructing external sounds. We apply this technology to robotic surgery to provide the surgeon with force feedback information with minimal distraction. The device is evaluated by pairing it with a force sensor that is attached to a suture pad. Four participants were tasked to complete 25 sutures on the suture pad while either receiving no feedback or audio, visual, or combined feedback that represents the magnitude of their applied force. Trials performed with bone conducting headphones had noticeable improvements compared to previous trials without feedback, while the most noticeable improvements were observed for cases with both visual and auditory feedback. Auditory feedback may have an important role in a robotic surgery setting and bone conduction headphones may enable this form of feedback with minimal distraction.

Biaxial sensing suture breakage warning system for robotic surgery

The number of procedures performed with robotic surgery may exceed one million globally in 2018. The continual lack of haptic feedback, however, forces surgeons to rely on visual cues in order to avoid breaking sutures due to excessive applied force. To mitigate this problem, the authors developed and validated a novel grasper-integrated system with biaxial shear sensing and haptic feedback to warn the operator prior to anticipated suture breakage. Furthermore, the design enables facile suture manipulation without a degradation in efficacy, as determined via measured tightness of resulting suture knots. Biaxial shear sensors were integrated with a da Vinci robotic surgical system. Novice subjects (n = 17) were instructed to tighten 10 knots, five times with the Haptic Feedback System (HFS) enabled, five times with the system disabled. Seven suture failures occurred in trials with HFS enabled while seventeen occurred in trials without feedback. The biaxial shear sensing system reduced the incidence of suture failure by 59% (p = 0.0371). It also resulted in 25% lower average applied force in comparison to trials without feedback (p = 0.00034), which is relevant because average force was observed to play a role in suture breakage (p = 0.03925). An observed 55% decrease in standard deviation of knot quality when using the HFS also indicates an improvement in consistency when using the feedback system. These results suggest this system may improve outcomes related to knot tying tasks in robotic surgery and reduce instances of suture failure while not degrading the quality of knots produced.

Suture Breakage Warning System for Robotic Surgery

As robotic surgery has increased in popularity, the lack of haptic feedback has become a growing issue due to the application of excessive forces that may lead to clinical problems such as intraoperative and postoperative suture breakage. Previous suture breakage warning systems have largely depended on visual and/or auditory feedback modalities, which have been shown to increase cognitive load and reduce operator performance. This work catalogues a new sensing technology and haptic feedback system (HFS) that can reduce instances of suture failure without negatively impacting performance outcomes including knot quality. Suture breakage is common in knot-tying as the pulling motion introduces prominent shear forces. A shear sensor mountable on the da Vinci robotic surgical system's Cadiere grasper detects forces that correlate to the suture's internal tension. HFS then provides vibration feedback to the operator as forces near a particular material's failure load. To validate the system, subjects tightened a total of four knots, two with the Haptic Feedback System (HFS) and two without feedback. The number of suture breakages were recorded and knot fidelity was evaluated by measuring knot slippage. Results showed that instances of suture failure were significantly reduced when HFS was enabled (p = 0.0078). Notably, knots tied with HFS also showed improved quality compared to those tied without feedback (p = 0.010). The results highlight the value of HFS in improving robotic procedure outcomes by reducing instances of suture failures, producing better knots, and reducing the need for corrective measures.

Optimal design and evaluation of a dexterous 4 DoFs haptic device based on delta architecture

This paper introduces a novel kinematic of a four degrees of freedom (DoFs) device based on Delta architecture. This new device is expected to be used as a haptic device for tele-operation applications. The challenging task was to obtain orientation DoFs from the Delta structure. A fourth leg is added to the Delta structure to convert translations into rotations and to provide translation of the handle. The fourth leg is linked to the base and to the moving platform by two universal joints. The architecture as well as the kinematic model of the new structure, called 4haptic, are presented. Comparisons in terms of kinematic behavior between the 4haptic device and the existing device developed based on spherical parallel manipulator architecture are presented. The results prove the improved behavior of the 4haptic device offering a singularity-free useful workspace, which makes it a suitable candidate to tele-operated system for Minimally Invasive Surgery. The dimensions of the 4haptic device, having the smallest workspace containing a prespecified region in space, are identified based on an optimal dimensional synthesis method.

The future of robotic surgery

How robotics could help shape the future of surgical care.

[Robotic Surgery - Who is The Boss?]

In the head and neck region, great potential is seen in robot-assisted surgery (RAS). Mainly in cancer surgery, the use of robotic systems seems to be of interest. Until today, two robotic systems (DaVinci® und FLEX®) have gained approval for clinical use in the head and neck region, and multiple other systems are currently in pre-clinical testing. Although, certain groups of patients may benefit from RAS, no unbiased randomized clinical studies are available. Until today, it was not possible to satisfactorily prove any advantage of RAS as compared to standard procedures. The limited clinical benefit and the additional financial burden seem to be the main reasons, why the comprehensive application of RAS has not been realized so far.This review article describes the large variety of clinical applications for RAS in the head and neck region. In addition, the financial and technical challenges, as well as ongoing developments of RAS are highlighted. Special focus is put on risks associated with RAS and current clinical studies. We believe, that RAS will find its way into clinical routine during the next years. Therefore, medical staff will have to increasingly face the technical, scientific and ethical features of RAS.

Prevalence of haptic feedback in robot-mediated surgery: a systematic review of literature

With the successful uptake and inclusion of robotic systems in minimally invasive surgery and with the increasing application of robotic surgery (RS) in numerous surgical specialities worldwide, there is now a need to develop and enhance the technology further. One such improvement is the implementation and amalgamation of haptic feedback technology into RS which will permit the operating surgeon on the console to receive haptic information on the type of tissue being operated on. The main advantage of using this is to allow the operating surgeon to feel and control the amount of force applied to different tissues during surgery thus minimising the risk of tissue damage due to both the direct and indirect effects of excessive tissue force or tension being applied during RS. We performed a two-rater systematic review to identify the latest developments and potential avenues of improving technology in the application and implementation of haptic feedback technology to the operating surgeon on the console during RS. This review provides a summary of technological enhancements in RS, considering different stages of work, from proof of concept to cadaver tissue testing, surgery in animals, and finally real implementation in surgical practice. We identify that at the time of this review, while there is a unanimous agreement regarding need for haptic and tactile feedback, there are no solutions or products available that address this need. There is a scope and need for new developments in haptic augmentation for robot-mediated surgery with the aim of improving patient care and robotic surgical technology further.

External force estimation and implementation in robotically assisted minimally invasive surgery

BACKGROUND

Robotically assisted minimally invasive surgery can offer many benefits over open surgery and laparoscopic minimally invasive surgery. However, currently, there is no force sensing and force feedback.

METHODS

This research was implemented using the da Vinci research kit. An external force estimation and implementation method was proposed based on dynamics and motor currents. The dynamics of the Patient Side Manipulator was modeled. The dynamic model was linearly parameterized. The estimation principle of external force was derived. The dynamic parameters were experimentally identified using a least squares method.

RESULTS

Several experiments including dynamic parameter identification, joint torque estimation, and external force estimation were performed. The results showed that the proposed method could implement force estimation without using a force sensor.

CONCLUSIONS

The force estimation method was proposed and implemented and experimental results showed the method worked and was feasible. This method could be used for force sensing in minimally invasive surgical robotics in the future.

The Role of Direct and Visual Force Feedback in Suturing Using a 7-DOF Dual-Arm Teleoperated System

The lack of haptic feedback in robotics-assisted surgery can result in tissue damage or accidental tool-tissue hits. This paper focuses on exploring the effect of haptic feedback via direct force reflection and visual presentation of force magnitudes on performance during suturing in robotics-assisted minimally invasive surgery (RAMIS). For this purpose, a haptics-enabled dual-arm master-slave teleoperation system capable of measuring tool-tissue interaction forces in all seven Degrees-of-Freedom (DOFs) was used. Two suturing tasks, tissue puncturing and knot-tightening, were chosen to assess user skills when suturing on phantom tissue. Sixteen subjects participated in the trials and their performance was evaluated from various points of view: force consistency, number of accidental hits with tissue, amount of tissue damage, quality of the suture knot, and the time required to accomplish the task. According to the results, visual force feedback was not very useful during the tissue puncturing task as different users needed different amounts of force depending on the penetration of the needle into the tissue. Direct force feedback, however, was more useful for this task to apply less force and to minimize the amount of damage to the tissue. Statistical results also reveal that both visual and direct force feedback were required for effective knot tightening: direct force feedback could reduce the number of accidental hits with the tissue and also the amount of tissue damage, while visual force feedback could help to securely tighten the suture knots and maintain force consistency among different trials/users. These results provide evidence of the importance of 7-DOF force reflection when performing complex tasks in a RAMIS setting.

Tensile strength and failure load of sutures for robotic surgery

BACKGROUND

Robotic surgical platforms have seen increased use among minimally invasive gastrointestinal surgeons (von Fraunhofer et al. in J Biomed Mater Res 19(5):595-600, 1985. doi: 10.1002/jbm.820190511 ). However, these systems still suffer from lack of haptic feedback, which results in exertion of excessive force, often leading to suture failures (Barbash et al. in Ann Surg 259(1):1-6, 2014. doi: 10.1097/SLA.0b013e3182a5c8b8 ). This work catalogs tensile strength and failure load among commonly used sutures in an effort to prevent robotic surgical consoles from exceeding identified thresholds. Trials were thus conducted on common sutures varying in material type, gauge size, rate of pulling force, and method of applied force.

METHODS

Polydioxanone, Silk, Vicryl, and Prolene, gauges 5-0 to 1-0, were pulled till failure using a commercial mechanical testing system. 2-0 and 3-0 sutures were further tested for the effect of pull rate on failure load at rates of 50, 200, and 400 mm/min. 3-0 sutures were also pulled till failure using a da Vinci robotic surgical system in unlooped, looped, and at the needle body arrangements.

RESULTS

Generally, Vicryl and PDS sutures had the highest mechanical strength (47-179 kN/cm²), while Silk had the lowest (40-106 kN/cm²). Larger diameter sutures withstand higher total force, but finer gauges consistently show higher force per unit area. The difference between material types becomes increasingly significant as the diameters decrease. Comparisons of identical suture materials and gauges show 27-50% improvement in the tensile strength over data obtained in 1985 (Ballantyne in Surg Endosc Other Interv Tech 16(10):1389-1402, 2002. doi: 10.1007/s00464-001-8283-7 ). No significant differences were observed when sutures were pulled at different rates. Reduction in suture strength appeared to be strongly affected by the technique used to manipulate the suture.

CONCLUSIONS

Availability of suture tensile strength and failure load data will help define software safety protocols for alerting a surgeon prior to suture failure during robotic surgery. Awareness of suture strength weakening with direct instrument manipulation may lead to the development of better techniques to further reduce intraoperative suture breakage.

Multi-Axis Force/Torque Sensor Based on Simply-Supported Beam and Optoelectronics

This paper presents a multi-axis force/torque sensor based on simply-supported beam and optoelectronic technology. The sensor's main advantages are: (1) Low power consumption; (2) low-level noise in comparison with conventional methods of force sensing (e.g., using strain gauges); (3) the ability to be embedded into different mechanical structures; (4) miniaturisation; (5) simple manufacture and customisation to fit a wide-range of robot systems; and (6) low-cost fabrication and assembly of sensor structure. For these reasons, the proposed multi-axis force/torque sensor can be used in a wide range of application areas including medical robotics, manufacturing, and areas involving human-robot interaction. This paper shows the application of our concept of a force/torque sensor to flexible continuum manipulators: A cylindrical MIS (Minimally Invasive Surgery) robot, and includes its design, fabrication, and evaluation tests.

The role of visual and direct force feedback in robotics-assisted mitral valve annuloplasty

BACKGROUND

The objective of this work was to determine the effect of both direct force feedback and visual force feedback on the amount of force applied to mitral valve tissue during ex vivo robotics-assisted mitral valve annuloplasty.

METHODS

A force feedback-enabled master-slave surgical system was developed to provide both visual and direct force feedback during robotics-assisted cardiac surgery. This system measured the amount of force applied by novice and expert surgeons to cardiac tissue during ex vivo mitral valve annuloplasty repair.

RESULTS

The addition of visual (2.16 ± 1.67), direct (1.62 ± 0.86), or both visual and direct force feedback (2.15 ± 1.08) resulted in lower mean maximum force applied to mitral valve tissue while suturing compared with no force feedback (3.34 ± 1.93 N; P < 0.05).

CONCLUSIONS

To achieve better control of interaction forces on cardiac tissue during robotics-assisted mitral valve annuloplasty suturing, force feedback may be required.