Feeling of pulsations in artificial arteries with a real time haptic feedback laparoscopic grasper: a validation study

INTRODUCTION

Despite the advancements in technology and organized training for surgeons in laparoscopic surgery, the persistent challenge of not being able to feel the resistance and characteristics of the tissue, including pulsations, remains unmet. A recently developed grasper (Optigrip®) with real time haptic feedback, based on photonic technology, aims to address this issue by restoring the tactile sensation for surgeons. The key question is whether pulsations can be detected and at what minimal size level they become clinical significant.

METHODS

To simulate arterial conditions during laparoscopic procedures, four different silicone tubes were created, representing the most prevalent arteries. These tubes were connected to a validated pressure system, generating a natural pulse ranging between 80 and 120 mm Hg. One control tube without pressure was added. The surgeons had to grasp these tubes blindly with the conventional grasper or the haptic feedback grasper in a randomized order. They then indicated whether they felt the pressure or not and the percentage of correct answers was calculated.

RESULTS

The haptic grasper successfully detected 96% of all pulsations, while the conventional grasper could only detect 6%. When considering the size of the arteries, the Optigrip® identified pulsations in 100% the 4 and 5 mm arteries and 92% of the smallest arteries. The conventional grasper was only able to feel the smallest arteries in 8%. These differences were highly significant (p < 0.0001).

CONCLUSION

This study demonstrated that the newly developed haptic feedback grasper enables detection of arterial pulsations during laparoscopy, filling an important absence in tactile perception within laparoscopic surgery.

Vision-based estimation of manipulation forces by deep learning of laparoscopic surgical images obtained in a porcine excised kidney experiment

In robot-assisted surgery, in which haptics should be absent, surgeons experience haptics-like sensations as "pseudo-haptic feedback". As surgeons who routinely perform robot-assisted laparoscopic surgery, we wondered if we could make these "pseudo-haptics" explicit to surgeons. Therefore, we created a simulation model that estimates manipulation forces using only visual images in surgery. This study aimed to achieve vision-based estimations of the magnitude of forces during forceps manipulation of organs. We also attempted to detect over-force, exceeding the threshold of safe manipulation. We created a sensor forceps that can detect precise pressure at the tips with three vectors. Using an endoscopic system that is used in actual surgery, images of the manipulation of excised pig kidneys were recorded with synchronized force data. A force estimation model was then created using deep learning. Effective detection of over-force was achieved if the region of the visual images was restricted by the region of interest around the tips of the forceps. In this paper, we emphasize the importance of limiting the region of interest in vision-based force estimation tasks.

SurgNet: Self-Supervised Pretraining With Semantic Consistency for Vessel and Instrument Segmentation in Surgical Images

Blood vessel and surgical instrument segmentation is a fundamental technique for robot-assisted surgical navigation. Despite the significant progress in natural image segmentation, surgical image-based vessel and instrument segmentation are rarely studied. In this work, we propose a novel self-supervised pretraining method (SurgNet) that can effectively learn representative vessel and instrument features from unlabeled surgical images. As a result, it allows for precise and efficient segmentation of vessels and instruments with only a small amount of labeled data. Specifically, we first construct a region adjacency graph (RAG) based on local semantic consistency in unlabeled surgical images and use it as a self-supervision signal for pseudo-mask segmentation. We then use the pseudo-mask to perform guided masked image modeling (GMIM) to learn representations that integrate structural information of intraoperative objectives more effectively. Our pretrained model, paired with various segmentation methods, can be applied to perform vessel and instrument segmentation accurately using limited labeled data for fine-tuning. We build an Intraoperative Vessel and Instrument Segmentation (IVIS) dataset, comprised of ~3 million unlabeled images and over 4,000 labeled images with manual vessel and instrument annotations to evaluate the effectiveness of our self-supervised pretraining method. We also evaluated the generalizability of our method to similar tasks using two public datasets. The results demonstrate that our approach outperforms the current state-of-the-art (SOTA) self-supervised representation learning methods in various surgical image segmentation tasks.

Sense of Agency and Skills Learning in Virtual-Mediated Environment: A Systematic Review

Agency is central to remote actions, and it may enhance skills learning due to a partial overlap between brain structures and networks, the promotion of confidence towards a telemanipulator, and the feeling of congruence of the motor choice to the motor plan. We systematically reviewed studies aiming to verify the role of agency in improving learning. Fifteen studies were selected from MEDLINE and Scopus®. When a mismatch is introduced between observed and performed actions, the decrease in agency and learning is proportional to the intensity of the mismatch, which is due to greater interference with the motor programming. Thanks to multisensory integration, agency and learning benefit both from sensory and performance feedback and from the timing of feedback based on control at the goal level or the perceptual-motor level. This work constitutes a bedrock for professional teleoperation settings (e.g., robotic surgery), with particular reference to the role of agency in performing complex tasks with remote control.

The benefits of haptic feedback in robot assisted surgery and their moderators: a meta-analysis

Robot assisted surgery (RAS) provides medical practitioners with valuable tools, decreasing strain during surgery and leading to better patient outcomes. While the loss of haptic sensation is a commonly cited disadvantage of RAS, new systems aim to address this problem by providing artificial haptic feedback. N = 56 papers that compared robotic surgery systems with and without haptic feedback were analyzed to quantify the performance benefits of restoring the haptic modality. Additionally, this study identifies factors moderating the effect of restoring haptic sensation. Overall results showed haptic feedback was effective in reducing average forces (Hedges' g = 0.83) and peak forces (Hedges' g = 0.69) applied during surgery, as well as reducing the completion time (Hedges' g = 0.83). Haptic feedback has also been found to lead to higher accuracy (Hedges' g = 1.50) and success rates (Hedges' g = 0.80) during surgical tasks. Effect sizes on several measures varied between tasks, the type of provided feedback, and the subjects' levels of surgical expertise, with higher levels of expertise generally associated with smaller effect sizes. No significant differences were found between virtual fixtures and rendering contact forces. Implications for future research are discussed.

Sensory substitution increases robotic surgical performance and sets the ground for a mediating role of the sense of embodiment: a systematic review

Sensory Substitution (SS) allows the elaboration of information via non preferential sensory modalities. This phenomenon occurs in robotic-assisted surgery (RAS), in which haptic feedback is lacking. It has been suggested that SS could sustain surgeons' proficiency by means of visual clues for inferring tactile information, that also promotes the feeling of haptic phantom sensations. A critical role in reaching a good performance in procedural tasks is also sustained by the Sense of Embodiment (SE), that is, the capacity to integrate objects into subjective bodily self-representation. As SE is enhanced by haptic sensations, we hypothesize a role of SS in promoting SE in RAS. Accordingly, the goal of this systematic review is to summarize the evidence pertaining the study of SS in RAS in order to highlight the impact on the performance, and to identify a mediating role of the SE in increasing dexterity in RAS. Eight studies selected from the MEDLINE and Scopus® databases met inclusion criteria for a qualitative synthesis. Results indicated that haptic to other modalities SS enhanced force consistency and accuracy, and decreased surgeon fatigue. Expert surgeons, as compared to novices, showed a better natural SS processing, testified by a proficient performance with and without SS aids. No studies investigated the mediating role of SE. These findings indicate that SS is subjected to learning and memory processes that help surgeons to rapidly derive haptic-correlates from visual clues, which are highly required for a good performance. Also, the higher ability of doing SS and the associated perception of haptic sensations might increase multisensory integration, which might sustain performance.

Microsurgery Robots: Applications, Design, and Development

Microsurgical techniques have been widely utilized in various surgical specialties, such as ophthalmology, neurosurgery, and otolaryngology, which require intricate and precise surgical tool manipulation on a small scale. In microsurgery, operations on delicate vessels or tissues require high standards in surgeons' skills. This exceptionally high requirement in skills leads to a steep learning curve and lengthy training before the surgeons can perform microsurgical procedures with quality outcomes. The microsurgery robot (MSR), which can improve surgeons' operation skills through various functions, has received extensive research attention in the past three decades. There have been many review papers summarizing the research on MSR for specific surgical specialties. However, an in-depth review of the relevant technologies used in MSR systems is limited in the literature. This review details the technical challenges in microsurgery, and systematically summarizes the key technologies in MSR with a developmental perspective from the basic structural mechanism design, to the perception and human-machine interaction methods, and further to the ability in achieving a certain level of autonomy. By presenting and comparing the methods and technologies in this cutting-edge research, this paper aims to provide readers with a comprehensive understanding of the current state of MSR research and identify potential directions for future development in MSR.

Surgery in the Next Space Missions

In the coming years, missions to the Moon and Mars shall be the new goals of space flight. The complexity of these missions due to the great distance from Earth and the unforeseen obstacles to settle on another planet have given rise to great concerns for crew health and survival. The need for advanced crew autonomy and a different approach to surgical emergency require new protocols and devices to help future crew medical officers and other crew members in a task of unprecedented difficulty. Hence, the increasing variety of schedules, devices, and protocols being developed. A serious health problem, such as an emerging surgical disease or severe trauma, can jeopardize the mission and survival of the entire crew. Many other difficulties are present in deep-space missions or settlements on other planets, such as communication and supply, also medical, delays, and shortage, and the presence of radiation. Progress in advanced technologies as well as the evolution of robotic surgery and the use of artificial intelligence are other topics of this review. In this particular area of research, even if we are still very far from an "intelligent robot", this evolution must be evaluated in the light of legislative and ethical considerations. This topic was presented at the annual meeting of the American College of Surgeons-Italy Chapter in 2021.

A Modular 3-Degrees-of-Freedom Force Sensor for Robot-Assisted Minimally Invasive Surgery Research

Effective force modulation during tissue manipulation is important for ensuring safe, robot-assisted, minimally invasive surgery (RMIS). Strict requirements for in vivo applications have led to prior sensor designs that trade off ease of manufacture and integration against force measurement accuracy along the tool axis. Due to this trade-off, there are no commercial, off-the-shelf, 3-degrees-of-freedom (3DoF) force sensors for RMIS available to researchers. This makes it challenging to develop new approaches to indirect sensing and haptic feedback for bimanual telesurgical manipulation. We present a modular 3DoF force sensor that integrates easily with an existing RMIS tool. We achieve this by relaxing biocompatibility and sterilizability requirements and by using commercial load cells and common electromechanical fabrication techniques. The sensor has a range of ±5 N axially and ±3 N laterally with errors of below 0.15 N and maximum errors below 11% of the sensing range in all directions. During telemanipulation, a pair of jaw-mounted sensors achieved average errors below 0.15 N in all directions. It achieved an average grip force error of 0.156 N. The sensor is for bimanual haptic feedback and robotic force control in delicate tissue telemanipulation. As an open-source design, the sensors can be adapted to suit other non-RMIS robotic applications.

A 3D-Printed Soft Haptic Device with Built-in Force Sensing Delivering Bio-Mimicked Feedback

Haptics plays a significant role not only in the rehabilitation of neurological disorders, such as stroke, by substituting necessary cognitive information but also in human-computer interfaces (HCIs), which are now an integral part of the recently launched metaverse. This study proposes a unique, soft, monolithic haptic feedback device (SoHapS) that was directly manufactured using a low-cost and open-source fused deposition modeling (FDM) 3D printer by employing a combination of soft conductive and nonconductive thermoplastic polyurethane (TPU) materials (NinjaTek, USA). SoHapS consists of a soft bellow actuator and a soft resistive force sensor, which are optimized using finite element modeling (FEM). SoHapS was characterized both mechanically and electrically to assess its performance, and a dynamic model was developed to predict its force output with given pressure inputs. We demonstrated the efficacy of SoHapS in substituting biofeedback with tactile feedback, such as gripping force, and proprioceptive feedback, such as finger flexion-extension positions, in the context of teleoperation. With its intrinsic properties, SoHapS can be integrated into rehabilitation robots and robotic prostheses, as well as augmented, virtual, and mixed reality (AR/VR/MR) systems, to induce various types of bio-mimicked feedback.

Vibro-Acoustic Sensing of Instrument Interactions as a Potential Source of Texture-Related Information in Robotic Palpation

The direct tactile assessment of surface textures during palpation is an essential component of open surgery that is impeded in minimally invasive and robot-assisted surgery. When indirectly palpating with a surgical instrument, the structural vibrations from this interaction contain tactile information that can be extracted and analysed. This study investigates the influence of the parameters contact angle α and velocity v→ on the vibro-acoustic signals from this indirect palpation. A 7-DOF robotic arm, a standard surgical instrument, and a vibration measurement system were used to palpate three different materials with varying α and v→. The signals were processed based on continuous wavelet transformation. They showed material-specific signatures in the time-frequency domain that retained their general characteristic for varying α and v→. Energy-related and statistical features were extracted, and supervised classification was performed, where the testing data comprised only signals acquired with different palpation parameters than for training data. The classifiers support vector machine and k-nearest neighbours provided 99.67% and 96.00% accuracy for the differentiation of the materials. The results indicate the robustness of the features against variations in the palpation parameters. This is a prerequisite for an application in minimally invasive surgery but needs to be confirmed in realistic experiments with biological tissues.

A Force-Feedback Methodology for Teleoperated Suturing Task in Robotic-Assisted Minimally Invasive Surgery

With robotic-assisted minimally invasive surgery (RAMIS), patients and surgeons benefit from a reduced incision size and dexterous instruments. However, current robotic surgery platforms lack haptic feedback, which is an essential element of safe operation. Moreover, teleportation control challenges make complex surgical tasks like suturing more time-consuming than those that use manual tools. This paper presents a new force-sensing instrument that semi-automates the suturing task and facilitates teleoperated robotic manipulation. In order to generate the ideal needle insertion trajectory and pass the needle through its curvature, the end-effector mechanism has a rotating degree of freedom. Impedance control was used to provide sensory information about needle-tissue interaction forces to the operator using an indirect force estimation approach based on data-based models. The operator's motion commands were then regulated using a hyperplanar virtual fixture (VF) designed to maintain the desired distance between the end-effector and tissue surface while avoiding unwanted contact. To construct the geometry of the VF, an optoelectronic sensor-based approach was developed. Based on the experimental investigation of the hyperplane VF methodology, improved needle-tissue interaction force, manipulation accuracy, and task completion times were demonstrated. Finally, experimental validation of the trained force estimation models and the perceived interaction forces by the user was conducted using online data, demonstrating the potential of the developed approach in improving task performance.

Design and realization of a novel haptic graspable interface for augmenting touch sensations

A novel haptic grasper that renders touch sensations to the user in 3-DoF (degrees of freedom), namely linear, rotary, and grasping motions, is presented. The touch sensations of the grasper include the combination of kinesthetic and tactile modalities such as stiffness, texture, and shape. The device is equipped with two swappable modular segments that provide stiffness and shape sensations. To increase the haptic fidelity, the textural surfaces that surround the outer surface of the segments are equipped with vibro-actuators underneath them. These vibro-actuators contribute to increasing the number of perceivable textures by varying amplitude, frequency, duration, and envelope of vibrations. The proposed device is characterized in terms of stiffness, shape and texture rendering capabilities. The experimental results validate the effectiveness of the developed haptic grasper in virtual/remote interactions. Also, the user studies and statistical analysis demonstrate that the users could perceive the high-fidelity haptic feedback with the unified sensations of kinesthetic and tactile cues.

A force-sensing retractor for robot-assisted transoral surgery

PURPOSE

In robot-assisted transoral surgery, frequent retraction operations are essential to leave space for the surgical procedure. Commercial clinical retractors are simply composed of mechanical parts and cannot sense the touching force.

METHODS

We propose a new retractor for robot-assisted transoral surgery. It supports sensing of the touching force when retracting the tissues. By designing the structure of the force sensors based on small piezoresistive elements, we build a sensory system that is well integrated with the retractor for transoral surgery. After calibration of the system, a simple equation is computed to decode the resultant force as well as the center of the contact location.

RESULTS

A standard measuring test is designed for the force-sensing retractor. The result shows that the measured force is up to 15 N, and the sensed force precision reaches 0.08 N with a sampling rate of 98 Hz. The dimensions of the sensory system fit the retractor well.

CONCLUSION

The experimental results demonstrate the potential of the proposed retractor in robot-assisted surgery. The retractor supports the provision of force feedback in an interactive manipulation mode and produces haptic information for the remote side in a teleoperated surgical robot system.

Smooth and safe tram journeys: tram driver perspectives and opportunities using a haptic master controller in a virtual reality environment

Tram drivers operate a master controller to control the acceleration and braking of the tram. Operation should ensure passenger comfort and safety through smooth tram motion and the avoidance of jerkiness that may cause passengers to fall in the carriage. This work investigates current driver practices and strategies for tram driving in normal operations through interviews and the capacity of a new haptic master controller to support drivers in achieving smooth and safe tram journeys. A haptic feedback algorithm based on viscosity was implemented on the master controller to provide drivers with feedback on the rate at which they were accelerating and braking the tram. This aspect was tested in a virtual tram within a simulated inner city virtual reality environment. Results indicate that the haptic master controller and coupled viscosity feedback algorithm did not increase smoothness of driving during the simulated experiences. Despite this, the drivers indicated a preference for the provision of further haptic information to support driving tasks and the overall journey safety and smoothness. Practitioner Summary: This research comprises two studies. The first investigates strategies currently used by drivers to operate a tram smoothly in order to elicit design requirements for a haptic tram master controller. The second study evaluates the impact of a novel haptic master controller on journey smoothness within a virtual environment.

Cardiac X-ray image-based haptic guidance for robot-assisted coronary intervention: a feasibility study

PURPOSE

Effective and efficient haptic guidance is desirable for tele-operated robotic surgery because it has a potential to enhance surgeon's skills, especially in coronary interventions where surgeon loses both an eye-hand coordination and a direct sight to the organ. This paper proposes a novel haptic guidance procedure-both kinesthetic and cutaneous, which solely depends upon X-ray images, for tele-robotic system that assists an efficient navigation of the guidewire towards the target location during a coronary intervention.

METHODS

Proposed methodology requires cardiologists to draw virtual fixtures (VFs) on angiograms as a preoperative procedure. During an operation, these VFs direct the guidewire to the desired coronary vessel. For this, the position and orientation of guidewire tip are calculated with respect to VFs' anatomy, using image processing on the real-time 2D fluoroscopic images. The haptic feedbacks are then rendered on to the master device depending on the interaction with attractive and repulsive, guidance and forbidden region VFs.

RESULTS

A feasibility study in the laboratory environment is performed by using a webcam as an image acquisition device and a phantom-based coronary vessel model. The subsequent statistical analysis shows that, on an average, a decrease of approx. 37% in task completion time is observed with haptic feedback. Moreover, haptic guidance is found effective for most difficult branch, whereas there is a minimal significance of such haptics for the easiest branch.

CONCLUSIONS

The proposed haptic guidance procedure may assist cardiologists for an efficient and effective guidewire navigation during a surgical procedure. The cutaneous haptics (vibration feedback) is found more helpful in coronary interventions compared with kinesthetic haptics (force feedback).

Development and Validation of a Mixed Reality Configuration of a Simulator for a Minimally Invasive Spine Surgery Using the Workspace of a Haptic Device and Simulator Users

Most surgical simulators leverage virtual or bench models to simulate reality. This study proposes and validates a method for workspace configuration of a surgical simulator which utilizes a haptic device for interaction with a virtual model and a bench model to provide additional tactile feedback based on planned surgical manoeuvers. Numerical analyses were completed to determine the workspace and position of a haptic device, relative to the bench model, used in the surgical simulator, and the determined configuration was validated using device limitations and user data from surgical and nonsurgical users. For the validation, surgeons performed an identical surgery on a cadaver prior to using the simulator, and their trajectories were then compared to the determined workspace for the haptic device. The configuration of the simulator was determined appropriate through workspace analysis and the collected user trajectories. Statistical analyses suggest differences in trajectories between the participating surgeons which were not affected by the imposed haptic workspace. This study, therefore, demonstrates a method to optimally position a haptic device with respect to a bench model while meeting the manoeuverability needs of a surgical procedure. The validation method identified workspace position and user trajectory towards ideal configuration of a mixed reality simulator.

Adaptive Surgical Robotic Training Using Real-Time Stylistic Behavior Feedback Through Haptic Cues

Surgical skill directly affects surgical procedure outcomes; thus, effective training is needed to ensure satisfactory results. Many objective assessment metrics have been developed that provide the trainee with descriptive feedback about their performance however, often lack feedback on how to improve performance. The most effective training method is one that is intuitive, easy to understand, personalized to the user,and provided in a timely manner. We propose a framework to enable user-adaptive training using near real-time detection of performance, based on intuitive styles of surgical movements, and design a haptic feedback framework to assist with correcting styles of movement. We evaluate the ability of three types of force feedback (spring, damping, and spring plus damping feedback), computed based on prior user positions, to improve different stylistic behaviors of the user during kinematically constrained reaching movement tasks. The results indicate that five out of six styles studied here were improved using at least one of the three types of force feedback. Task performance metrics were compared in the presence of the three types of feedback. Task time was statistically significantly lower when applying spring feedback, compared to the other two types of feedback. Path straightness and targeting error were statistically significantly improved when using spring-damping feedback compared to the other two types of feedback. This study presents a groundwork for adaptive training in robotic surgery based on near real-time human-centric models of surgical behavior.

A learning-based tip contact force estimation method for tendon-driven continuum manipulator

Although tendon-driven continuum manipulators have been extensively researched, how to realize tip contact force sensing in a more general and efficient way without increasing the diameter is still a challenge. Rather than use a complex modeling approach, this paper proposes a general tip contact force-sensing method based on a recurrent neural network that takes the tendons’ position and tension as the input of a recurrent neural network and the tip contact force of the continuum manipulator as the output and fits this static model by means of machine learning so that it may be used as a real-time contact force estimator. We also designed and built a corresponding three-degree-of-freedom contact force data acquisition platform based on the structure of a continuum manipulator designed in our previous studies. After obtaining training data, we built and compared the performances of a multi-layer perceptron-based contact force estimator as a baseline and three typical recurrent neural network-based contact force estimators through TensorFlow framework to verify the feasibility of this method. We also proposed a manually decoupled sub-estimators algorithm and evaluated the advantages and disadvantages of those two methods.

Toward Mechanochromic Soft Material-Based Visual Feedback for Electronics-Free Surgical Effectors

A chromogenically reversible, mechanochromic pressure sensor is integrated into a mininvasive surgical grasper compatible with the da Vinci robotic surgical system. The sensorized effector, also featuring two soft-material jaws, encompasses a mechanochromic polymeric inset doped with functionalized spiropyran (SP) molecule, designed to activate mechanochromism at a chosen pressure and providing a reversible color change. Considering such tools are systematically in the visual field of the operator during surgery, color change of the mechanochromic effector can help avoid tissue damage. No electronics is required to control the devised visual feedback. SP-doping of polydimethylsiloxane (2.5:1 prepolymer/curing agent weight ratio) permits to modulate the mechanochromic activation pressure, with lower values around 1.17 MPa for a 2% wt. SP concentration, leading to a shorter chromogenic recovery time of 150 s at room temperature (25 °C) under green light illumination. Nearly three-times shorter recovery time is observed at body temperature (37 °C). To the best of knowledge, this study provides the first demonstration of mechanochromic materials in surgery, in particular to sensorize unpowered surgical effectors, by avoiding dramatic increases in tool complexity due to additional electronics, thus fostering their application. The proposed sensing strategy can be extended to further tools and scopes.

Enhancing the Localization of Uterine Leiomyomas Through Cutaneous Softness Rendering for Robot-Assisted Surgical Palpation Applications

Integrating tactile feedback for lump localization in Robot-assisted Minimally Invasive Surgery (RMIS) represents an open research issue, which is still far to be solved. Main reasons for this are related e.g. to the need for a transparent connection with the teleoperating console, and an intuitive decoding of the delivered information. In this article, we focus on the specific case of RMIS treatment of uterine leiomyomas or fibroids, where little has been done in haptics to improve the outcomes of robotics-enabled palpation tasks. In this article, we propose the usage of a wearable haptic interface for softness rendering as a lump display. The device was integrated in a teleoperation architecture that simulates a robot-assisted surgical palpation task of leiomyomas. This article moved from an ex-vivo sample characterization of uterine tissues to show the effectiveness of our interface in conveying meaningful softness information. We extensively tested our system with gynecologic surgeons in palpation tasks with silicone specimens, which replicated the characteristics of uterine tissues with embedded leyomiomas. Results show that our system enables a softness-based discrimination of the embedded fibroids comparable to the one that physicians would achieve using directly their fingers in palpation tasks. Furthermore, the feedback provided by the haptic interface was perceived as comfortable, intuitive, and highly useful for fibroid localization.

Robotic Transanal Total Mesorectal Excision (RTaTME): State of the Art

Total mesorectal excision (TME) is the gold standard technique for the surgical management of rectal cancer. The transanal approach to the mesorectum was introduced to overcome the technical difficulties related to the distal rectal dissection. Since its inception, interest in transanal mesorectal excision has grown exponentially and it appears that the benefits are maximal in patients with mid-low rectal cancer where anatomical and pathological features represent the greatest challenges. Current evidence demonstrates that this approach is safe and feasible, with oncological and functional outcome comparable to conventional approaches, but with specific complications related to the technique. Robotics might potentially simplify the technical steps of distal rectal dissection, with a shorter learning curve compared to the laparoscopic transanal approach, but with higher costs. The objective of this review is to critically analyze the available literature concerning robotic transanal TME in order to define its role in the management of rectal cancer and to depict future perspectives in this field of research.

The Path to Surgical Robotics in Neurosurgery

Robotic systems may help efficiently execute complicated tasks that require a high degree of accuracy, and this, in large part, explains why robotics have garnered widespread use in a variety of neurosurgical applications, including intracranial biopsies, spinal instrumentation, and placement of intracranial leads. The use of robotics in neurosurgery confers many benefits, and inherent limitations, to both surgeons and their patients. In this narrative review, we provide a historical overview of robotics and its implementation across various surgical specialties, and discuss the various robotic systems that have been developed specifically for neurosurgical applications. We also discuss the relative advantages of robotic systems compared to traditional surgical techniques, particularly as it pertains to integration of image guidance with the ability of the robotic arm to reliably execute pre-planned tasks. As more neurosurgeons adopt the use of robotics in their practice, we postulate that further technological advancements will become available that will help achieve improved technical capabilities, user experience, and overall patient clinical outcomes.

The Functional Rotational Workspace of a Human-Robot System can be Influenced by Adjusting the Telemanipulator Handle Orientation

The handle design of telemanipulation master devices has not been extensively studied so far. However, the master device handle is an integral part of the robotic system through which the user interacts with the system. Previous work showed that the size and shape of the functional rotational workspace of the human-robot system and its usability are influenced by the design of the master device handle. Still, in certain situations, e.g., due to user preference, a specific grasp type handle might be desired. Therefore, in this article, we provide a systematic approach on how to assess and adjust the functional rotational workspace of a human-robot system. We investigated the functional rotational workspace with two exemplary grasp type handles and two different mounting orientations for each handle. The results showed that by adapting the handle orientation in the home configuration of the telemanipulator, the functional rotational workspace of the human-robot system can be adjusted systematically to cover more of the mechanical workspace of the master device. Finally, we deduct recommendations on how to choose and adjust a telemanipulator handle.

An Application-Based Review of Haptics Technology

Recent technological development has led to the invention of different designs of haptic devices, electromechanical devices that mediate communication between the user and the computer and allow users to manipulate objects in a virtual environment while receiving tactile feedback. The main criteria behind providing an interactive interface are to generate kinesthetic feedback and relay information actively from the haptic device. Sensors and feedback control apparatus are of paramount importance in designing and manufacturing a haptic device. In general, haptic technology can be implemented in different applications such as gaming, teleoperation, medical surgeries, augmented reality (AR), and virtual reality (VR) devices. This paper classifies the application of haptic devices based on the construction and functionality in various fields, followed by addressing major limitations related to haptics technology and discussing prospects of this technology.

Proposal and Evaluation of Visual Haptics for Manipulation of Remote Machine System

Remote machine systems have drawn a lot of attention owing to accelerations of virtual reality (VR), augmented reality (AR), and the fifth generation (5G) networks. Despite recent trends of developing autonomous systems, the realization of sophisticated dexterous hand that can fully replace human hands is considered to be decades away. It is also extremely difficult to reproduce the sensilla of complex human hands. On the other hand, it is known that humans can perceive haptic information from visual information even without any physical feedback as cross modal sensation between visual and haptics sensations or pseudo haptics. In this paper, we propose a visual haptic technology, where haptic information is visualized in more perceptual images overlaid at the contact points of a remote machine hand. The usability of the proposed visual haptics was evaluated by subject's brain waves aiming to find out a new approach for quantifying "sense of oneness." In our proof-of-concept experiments using VR, subjects are asked to operate a virtual arm and hand presented in the VR space, and the performance of the operation with and without visual haptics information as measured with brain wave sensing. Consequently, three results were verified. Firstly, the information flow in the brain were significantly reduced with the proposed visual haptics for the whole α, β, and θ-waves by 45% across nine subjects. This result suggests that superimposing visual effects may be able to reduce the cognitive burden on the operator during the manipulation for the remote machine system. Secondly, high correlation (Pearson correlation factor of 0.795 at a p-value of 0.011) was verified between the subjective usability points and the brainwave measurement results. Finally, the number of the task successes across sessions were improved in the presence of overlaid visual stimulus. It implies that the visual haptics image could also facilitate operators' pre-training to get skillful at manipulating the remote machine interface more quickly.

Joint-level force sensing for indirect hybrid force/position control of continuum robots with friction

Continuum robots offer the dexterity and obstacle circumvention capabilities necessary to enable surgery in deep surgical sites. They also can enable joint-level ex situ force sensing (JEFS), which provides an estimate of end-effector wrenches given joint-level forces. Prior works on JEFS relied on a restrictive embodiment with minimal actuation line friction and captured model and frictional actuation transmission uncertainties using a configuration space formulation. In this work, we overcome these limitations. First, frictional losses are canceled using a feed-forward term based on support vector regression in joint space. Then, regression maps and their interpolation are used to account for actuation hysteresis. The residual joint-force error is then further minimized using a least-squares model parameter update. An indirect hybrid force/position controller using JEFS is presented with evaluation carried out on a realistic pre-clinically deployable insertable robotic effectors platform (IREP) for single-port access surgery. Automated mock force-controlled ablation, exploration, and knot tightening are evaluated. A user study involving the daVinci Research Kit surgeon console and the IREP as a surgical slave was carried out to compare the performance of users with and without force feedback based on JEFS for force-controlled ablation and knot tightening. Results in automated experiments and a user study of telemanipulated experiments suggest that intrinsic force-sensing can achieve levels of force uncertainty and force regulation errors of the order of 0.2 N. Using JEFS and automated task execution, repeatability, and force regulation accuracy is shown to be comparable to using a commercial force sensor for human-in-the-loop feedback.

Telelocomotion—Remotely Operated Legged Robots

Teleoperated systems enable human control of robotic proxies and are particularly amenable to inaccessible environments unsuitable for autonomy. Examples include emergency response, underwater manipulation, and robot assisted minimally invasive surgery. However, teleoperation architectures have been predominantly employed in manipulation tasks, and are thus only useful when the robot is within reach of the task. This work introduces the idea of extending teleoperation to enable online human remote control of legged robots, or telelocomotion, to traverse challenging terrain. Traversing unpredictable terrain remains a challenge for autonomous legged locomotion, as demonstrated by robots commonly falling in high-profile robotics contests. Telelocomotion can reduce the risk of mission failure by leveraging the high-level understanding of human operators to command in real-time the gaits of legged robots. In this work, a haptic telelocomotion interface was developed. Two within-user studies validate the proof-of-concept interface: (i) The first compared basic interfaces with the haptic interface for control of a simulated hexapedal robot in various levels of traversal complexity; (ii) the second presents a physical implementation and investigated the efficacy of the proposed haptic virtual fixtures. Results are promising to the use of haptic feedback for telelocomotion for complex traversal tasks.

Review of Advanced Medical Telerobots

The advent of telerobotic systems has revolutionized various aspects of the industry and human life. This technology is designed to augment human sensorimotor capabilities to extend them beyond natural competence. Classic examples are space and underwater applications when distance and access are the two major physical barriers to be combated with this technology. In modern examples, telerobotic systems have been used in several clinical applications, including teleoperated surgery and telerehabilitation. In this regard, there has been a significant amount of research and development due to the major benefits in terms of medical outcomes. Recently telerobotic systems are combined with advanced artificial intelligence modules to better share the agency with the operator and open new doors of medical automation. In this review paper, we have provided a comprehensive analysis of the literature considering various topologies of telerobotic systems in the medical domain while shedding light on different levels of autonomy for this technology, starting from direct control, going up to command-tracking autonomous telerobots. Existing challenges, including instrumentation, transparency, autonomy, stochastic communication delays, and stability, in addition to the current direction of research related to benefit in telemedicine and medical automation, and future vision of this technology, are discussed in this review paper.

Overcoming the Arduous Transition for Robotic Hepatopancreatobiliary Cases: A Multi-Procedure Learning Curve Study Utilizing CUSUM Analysis

BACKGROUND/OBJECTIVE

Quick optimization and mastery of a new technique is an important part of procedural medicine, especially in the field of minimally invasive surgery. Complex surgeries such as robotic pancreaticoduodenectomies (RPD) and robotic distal pancreatectomies (RDP) have a steep learning curve; therefore, findings that can help expedite the burdensome learning process are extremely beneficial. This single-surgeon study aims to report the learning curves of RDP, RPD, and robotic Heller myotomy (RHM) and to review the results' implications for the current state of robotic hepatopancreaticobiliary (HPB) surgery.

STUDY DESIGN

This is a retrospective case series of a prospectively maintained database at a non-university tertiary care center. Total of 175 patients underwent either RDP, RPD, or RHM with the surgeon (DRJ) from January 2014 to January 2020.

RESULTS

Statistical significance of operating room time (ORT) was noted after 47 cases for RDP (p < 0.05), 51 cases for RPD (p < 0.0001), and 18 cases for RHM (p < 0.05). Mean ORT after the statistical mastery of the procedure for RDP, RPD, and RHM was 124, 232, 93 min, respectively. No statistical significance was noted for estimated blood loss or length of stay.

CONCLUSIONS

Robotic HPB procedures have significantly higher learning curves compared to non-HPB procedures, even for an experienced HPB surgeon with extensive laparoscopic experience. Our RPD curve, however, is quicker than the literature average. We suggest that this is because of the simultaneous implementation of HPB (RDP and RPD) and non-HPB robotic surgeries with a shorter learning curve-especially foregut procedures such as RHM-into an experienced surgeon's practice. This may accelerate the learning process without compromising patient safety and outcomes.

Characterizing Limits of Vision-Based Force Feedback in Simulated Surgical Tool-Tissue Interaction

Haptic feedback can render real-time force interactions with computer simulated objects. In several telerobotic applications, it is desired that a haptic simulation reflects a physical task space or interaction accurately. This is particularly true when excessive applied force can result in disastrous consequences, as with the case of robot-assisted minimally invasive surgery (RMIS) and tissue damage. Since force cannot be directly measured in RMIS, non-contact methods are desired. A promising direction of non-contact force estimation involves the primary use of vision sensors to estimate deformation. However, the required fidelity of non-contact force rendering of deformable interaction to maintain surgical operator performance is not well established. This work attempts to empirically evaluate the degree to which haptic feedback may deviate from ground truth yet result in acceptable teleoperated performance in a simulated RMIS-based palpation task. A preliminary user-study is conducted to verify the utility of the simulation platform, and the results of this work have implications in haptic feedback for RMIS and inform guidelines for vision-based tool-tissue force estimation. An adaptive thresholding method is used to collect the minimum and maximum tolerable errors in force orientation and magnitude of presented haptic feedback to maintain sufficient performance.

MU-LapaRobot: A Corporative Surgical Robot for Laparoscopic Surgery

Objective:

Development of surgical instrument robot (MU-LapaRobot) for assisting in conventional laparoscopic surgery.

Methods:

A robot was designed based on instrument movement for a conventional procedure. The mechanism mimics and constrains manipulation movement through the wire-driven transmission. It is flexible for robot end-effector, which has a lightweight and small size. The usability of the robot is passive and active robot tasks with an interconnected driving system. Three main parts of the robot are robot end-effector, transmission, and a driving system.

Results:

On a robot platform, a parameter for setup is robot posture. The adjustment of robot position and projection of manipulation area is influenced by the accuracy of movement. To verify movement, command and exact movements are measured. Compensation with the control system is improved in order to improve the accuracy of the system.

Conclusion:

MU-LapaRobot provides surgical instrument manipulation by using wire-driven transmission with an effective system and requires less interference in the conventional operation.

Robotic Retinal Surgery Impacts on Scleral Forces: In Vivo Study

Purpose

This study aims to map force interaction between instrument and sclera of in vivo rabbits during retinal procedures, and verify if a robotic active force control could prevent unwanted increase of forces on the sclera.

Methods

Experiments consisted in the performance of intraocular movements of a force sensing instrument, adjacent to the retinal surface, in radial directions, from the center to the periphery and back, and compared manual manipulations with robotic assistance and also robotic assistance with an active force control. This protocol was approved by the Animal Use and Ethical Committee and experiments were according to ARVO Statement of Animal Use.

Results

Mean forces using manual manipulations were 115 ± 51 mN. Using robotic assistance, mean forces were 118 ± 49 mN. Using an active force control method, overall mean forces reduced to 69 ± 15, with a statistical difference compared with other methods (P < 0.001). Comparing intraocular directions, superior sector required higher forces and the force control method reduced differences in forces between users and retained the same force pattern between them.

Conclusions

Results validate that the introduction of robotic assistance might increase the dynamic interactions between instrument and sclera, and the addition of an active force control method reduces the forces at levels lower than manual manipulations.

Translational Relevance

All marketing benefits from extreme accuracy and stability from robots, however, redundancy of safety mechanisms during intraocular manipulations, especially on force control and surgical awareness, would allow all utility of robotic assistance in ophthalmology.

Conceptual design of force reflection control for teleoperated bone surgery

Bilateral control of teleoperated robots still poses a challenge, especially if environment properties vary over a large degree. Most currently available systems do not provide force feedback and consequently surgeons still have to estimate contact forces predominantly visually. During drilling or milling in bone surgery, visual estimation is virtually impossible due to hardly any deformations. However, the force progression contains important complimentary information for the surgeon. Therefore, a concept for a force-reflecting controller for drilling or milling during teleoperated bone surgery was developed and tested on a one degree of freedom (DOF) test setup. First, the desired behavior and control architectures were derived based on the context of bone surgery. The resulting controller combines three control architectures in a switching controller, depending on the tool actuation and environment properties. Experimental results with a 1-DOF test setup showed the desired control and switching behavior, while remaining stable. Therefore, the developed control concept seems promising for teleoperated bone surgery.

Haptic Intracorporeal Palpation Using a Cable-Driven Parallel Robot: A User Study

OBJECTIVE

Intraoperative palpation is a surgical gesture jeopardized by the lack of haptic feedback which affects robotic minimally invasive surgery. Restoring the force reflection in teleoperated systems may improve both surgeons' performance and procedures' outcome.

METHODS

A force-based sensing approach was developed, based on a cable-driven parallel manipulator with anticipated seamless and low-cost integration capabilities in teleoperated robotic surgery. No force sensor on the end-effector is used, but tissue probing forces are estimated from measured cable tensions. A user study involving surgical trainees (n = 22) was conducted to experimentally evaluate the platform in two palpation-based test-cases on silicone phantoms. Two modalities were compared: visual feedback alone and both visual + haptic feedbacks available at the master site.

RESULTS

Surgical trainees' preference for the modality providing both visual and haptic feedback is corroborated by both quantitative and qualitative metrics. Hard nodules detection sensitivity improves (94.35 ± 9.1% vs 76.09 ± 19.15% for visual feedback alone), while also exerting smaller forces (4.13 ± 1.02 N vs 4.82 ± 0.81 N for visual feedback alone) on the phantom tissues. At the same time, the subjective perceived workload decreases.

CONCLUSION

Tissue-probe contact forces are estimated in a low cost and unique way, without the need of force sensors on the end-effector. Haptics demonstrated an improvement in the tumor detection rate, a reduction of the probing forces, and a decrease in the perceived workload for the trainees.

SIGNIFICANCE

Relevant benefits are demonstrated from the usage of combined cable-driven parallel manipulators and haptics during robotic minimally invasive procedures. The translation of robotic intraoperative palpation to clinical practice could improve the detection and dissection of cancer nodules.

Impact of haptic feedback on applied intracorporeal forces using a novel surgical robotic system-a randomized cross-over study with novices in an experimental setup

Background

Most currently used surgical robots have no force feedback; the next generation displays forces visually. A novel single-port robotic surgical system called FLEXMIN has been developed. Through an outer diameter of 38 mm, two instruments are teleoperated from a surgeon’s control console including true haptic force feedback. One additional channel incorporates a telescope, another is free for special instrument functions.

Methods

This randomized cross-over study analyzed the effect of haptic feedback on the application of intracorporeal forces. In a standardized experiment setup, the subjects had to draw circles with the surgical robot as gently as possible. The applied forces, the required time spans, and predefined error rates were measured.

Results

Without haptic feedback, the maximum forces (median/IQR) were 6.43 N/2.96 N. With haptic feedback, the maximum forces were lower (3.57 N/1.94 N, p < 0.001). Also, the arithmetic means of the force progression (p < 0.001) and their standard deviations (p < 0.001) were lower. Not significant were the shorter durations and lower error rates. No sequence effect of force or duration was detected. No characteristic learning or fatigue curve was observed.

Conclusions

In the experiment setup, the true haptic force feedback can reduce the applied intracorporeal robotic force to one-half when considering the aspects maximum, means, and standard deviation. Other test tasks are needed to validate the influence of force feedback on surgical efficiency and safety.

A review of haptic feedback in tele-operated robotic surgery

During traditional surgery, the surgeons' hands are in direct contact with organs, and surgeons rely on the sense of touch to perform surgery. In teleoperated robotic systems, all physical connections between the surgeon and both the robot and patient, are absent. The surgeon must estimate the force exerted on organs, based only on visual deformation of tissues he is pulling, pushing, gripping, or suturing. It is hard to imagine how to operate with no haptic sensations, and it is surprising that commercially available robots didn't include until now any Haptic Feedback, despite reports about tissue injury, and inability to perform complex manipulation. The sense of touch must be created by stimuli sensed by the surgeon. Haptic sensors are required to collect and send haptic information, and display them on the operator's side, creating telepresence, known as transparency. Multiple ways have been developed to improve transparency through force feedback and tactile feedback. However, this interferes with the stability of the closed-loop controlling interactions between master, robot and remote environment. Cutaneous feedback is more stable and less transparent; force feedback is more transparent and less stable. Thus, multimodal platforms of haptic feedback would try to find the best trade-off between both modalities.

Toward Enhanced Teleoperation Through Embodiment

Telerobotics aims to transfer human manipulation skills and dexterity over an arbitrary distance and at an arbitrary scale to a remote workplace. A telerobotic system that is transparent enables a natural and intuitive interaction. We postulate that embodiment (with three sub-components: sense of ownership, agency, and self-location) of the robotic system leads to optimal perceptual transparency and increases task performance. However, this has not yet been investigated directly. We reason along four premises and present findings from the literature that substantiate each of them: (1) the brain can embody non-bodily objects (e.g., robotic hands), (2) embodiment can be elicited with mediated sensorimotor interaction, (3) embodiment is robust against inconsistencies between the robotic system and the operator's body, and (4) embodiment positively correlates to dexterous task performance. We use the predictive encoding theory as a framework to interpret and discuss the results reported in the literature. Numerous previous studies have shown that it is possible to induce embodiment over a wide range of virtual and real extracorporeal objects (including artificial limbs, avatars, and android robots) through mediated sensorimotor interaction. Also, embodiment can occur for non-human morphologies including for elongated arms and a tail. In accordance with the predictive encoding theory, none of the sensory modalities is critical in establishing ownership, and discrepancies in multisensory signals do not necessarily lead to loss of embodiment. However, large discrepancies in terms of multisensory synchrony or visual likeness can prohibit embodiment from occurring. The literature provides less extensive support for the link between embodiment and (dexterous) task performance. However, data gathered with prosthetic hands do indicate a positive correlation. We conclude that all four premises are supported by direct or indirect evidence in the literature, suggesting that embodiment of a remote manipulator may improve dexterous performance in telerobotics. This warrants further implementation testing of embodiment in telerobotics. We formulate a first set of guidelines to apply embodiment in telerobotics and identify some important research topics.

Surgeon-Centered Analysis of Robot-Assisted Needle Driving Under Different Force Feedback Conditions

Robotic assisted minimally invasive surgery (RAMIS) systems present many advantages to the surgeon and patient over open and standard laparoscopic surgery. However, haptic feedback, which is crucial for the success of many surgical procedures, is still an open challenge in RAMIS. Understanding the way that haptic feedback affects performance and learning can be useful in the development of haptic feedback algorithms and teleoperation control systems. In this study, we examined the performance and learning of inexperienced participants under different haptic feedback conditions in a task of surgical needle driving via a soft homogeneous deformable object-an artificial tissue. We designed an experimental setup to characterize their movement trajectories and the forces that they applied on the artificial tissue. Participants first performed the task in an open condition, with a standard surgical needle holder, followed by teleoperation in one of three feedback conditions: (1) no haptic feedback, (2) haptic feedback based on position exchange, and (3) haptic feedback based on direct recording from a force sensor, and then again with the open needle holder. To quantify the effect of different force feedback conditions on the quality of needle driving, we developed novel metrics that assess the kinematics of needle driving and the tissue interaction forces, and we combined our novel metrics with classical metrics. We analyzed the final teleoperated performance in each condition, the improvement during teleoperation, and the aftereffect of teleoperation on the performance when using the open needle driver. We found that there is no significant difference in the final performance and in the aftereffect between the 3 conditions. Only the two conditions with force feedback presented statistically significant improvement during teleoperation in several of the metrics, but when we compared directly between the improvements in the three different feedback conditions none of the effects reached statistical significance. We discuss possible explanations for the relative similarity in performance. We conclude that we developed several new metrics for the quality of surgical needle driving, but even with these detailed metrics, the advantage of state of the art force feedback methods to tasks that require interaction with homogeneous soft tissue is questionable.