Haptic interaction in robot-assisted endoscopic surgery: a sensorized end-effector

A Survey on Force Sensing Techniques in Robot-Assisted Minimally Invasive Surgery

Minimally invasive surgery (MIS) is commonly used in some robotic-assisted surgery (RAS) systems. However, many RAS lack the strength and tactile sensation of surgical tools. Therefore, researchers have developed various force sensing techniques in robot-assisted minimally invasive surgery (RMIS). This paper provides a systematic classification and review of force sensing approaches in the field of RMIS, with a particular focus on direct and indirect force sensing. In this survey, the relevant literature on various sensing principles, haptic sensor design standards, and sensing technologies between 2000 and 2022 is reviewed. This survey can also serve as a roadmap for future developments by identifying the shortcomings of the field and discussing the emerging trends in force sensing methods.

A 3D-Printed Soft Haptic Device with Built-in Force Sensing Delivering Bio-Mimicked Feedback

Haptics plays a significant role not only in the rehabilitation of neurological disorders, such as stroke, by substituting necessary cognitive information but also in human-computer interfaces (HCIs), which are now an integral part of the recently launched metaverse. This study proposes a unique, soft, monolithic haptic feedback device (SoHapS) that was directly manufactured using a low-cost and open-source fused deposition modeling (FDM) 3D printer by employing a combination of soft conductive and nonconductive thermoplastic polyurethane (TPU) materials (NinjaTek, USA). SoHapS consists of a soft bellow actuator and a soft resistive force sensor, which are optimized using finite element modeling (FEM). SoHapS was characterized both mechanically and electrically to assess its performance, and a dynamic model was developed to predict its force output with given pressure inputs. We demonstrated the efficacy of SoHapS in substituting biofeedback with tactile feedback, such as gripping force, and proprioceptive feedback, such as finger flexion-extension positions, in the context of teleoperation. With its intrinsic properties, SoHapS can be integrated into rehabilitation robots and robotic prostheses, as well as augmented, virtual, and mixed reality (AR/VR/MR) systems, to induce various types of bio-mimicked feedback.

Realization of Force Detection and Feedback Control for Slave Manipulator of Master/Slave Surgical Robot

Robot-assisted minimally invasive surgery (MIS) has received increasing attention, both in the academic field and clinical operation. Master/slave control is the most widely adopted manipulation mode for surgical robots. Thus, sensing the force of the surgical instruments located at the end of the slave manipulator through the main manipulator is critical to the operation. This study mainly addressed the force detection of the surgical instrument and force feedback control of the serial surgical robotic arm. A measurement device was developed to record the tool end force from the slave manipulator. An elastic element with an orthogonal beam structure was designed to sense the strain induced by force interactions. The relationship between the acting force and the output voltage was obtained through experiment, and the three-dimensional force output was decomposed using an extreme learning machine algorithm while considering the nonlinearity. The control of the force from the slave manipulator end was achieved. An impedance control strategy was adopted to restrict the force interaction amplitude. Modeling, simulation, and experimental verification were completed on the serial robotic manipulator platform along with virtual control in the MATLAB/Simulink software environment. The experimental results show that the measured force from the slave manipulator can provide feedback for impedance control with a delay of 0.15 s.

Design and implementation of a hand-held robot-assisted minimally invasive surgical device with enhanced intuitive manipulability and stable grip force

BACKGROUND

The conventional hand-held minimally invasive surgical devices commonly suffer from non-intuitive manipulability and restricted flexibility for operation.

METHODS

A hand-held surgical device with enhanced intuitive manipulability and stable grip force was proposed for minimally invasive surgery (MIS). The dexterous instrument and isomorphic handle were designed, and the cable transmission structure and model of the instrument were analysed. A modelling method for grip force pre-compensation was proposed to produce stable grip forces under different posture.

RESULTS

The prototype of the proposed MIS device was developed, and the related experiments were carried out. The maximum opening angle error was 1.2°. Compared with the non-compensation model, the variation of grip force reduced 8 times with the pre-compensation model. The animal vivo experiments verified the feasibility and practicability of the device.

CONCLUSIONS

The proposed hand-held device could provide intuitive manipulability and stable operation, which contributes to the performance improvement of the MIS.

Review of surgical robotic systems for keyhole and endoscopic procedures: state of the art and perspectives

Minimally invasive surgery, including laparoscopic and thoracoscopic procedures, benefits patients in terms of improved postoperative outcomes and short recovery time. The challenges in hand-eye coordination and manipulation dexterity during the aforementioned procedures have inspired an enormous wave of developments on surgical robotic systems to assist keyhole and endoscopic procedures in the past decades. This paper presents a systematic review of the state-of-the-art systems, picturing a detailed landscape of the system configurations, actuation schemes, and control approaches of the existing surgical robotic systems for keyhole and endoscopic procedures. The development challenges and future perspectives are discussed in depth to point out the need for new enabling technologies and inspire future researches.

An extended algorithm for autonomous grasping of soft tissues during robotic surgery

BACKGROUND

Autonomous grasping of soft tissues can facilitate the robotic surgery procedures. The previous attempts for implementing auto-grasping have been based on a simplistic representation of the actual surgery maneuvers.

METHOD

A generalized three-zone grasp model was introduced to consider the effect of the pull force angulation on the grasp mode, that is, damage, slip, or safe grasp. Also, an extended auto-grasping algorithm was proposed in which the trigger force is automatically controlled against the pull force magnitude and direction, to achieve a safe and secure grasp.

RESULTS

The autonomous grasping experiments against a varying pull force in a phantom study indicated a good agreement between the desired and actual pinch and trigger forces (root mean square errors lower than 0.168 N and 0.280 N, respectively) and no sign of tissue tear or slippage.

CONCLUSIONS

The proposed auto-grasping algorithm can help manipulating the soft tissues safely and effectively during robotic surgery procedures.

Pneumatically driven surgical forceps displaying a magnified grasping torque

BACKGROUND

Sensing the grasping force and displaying the force for the operator are important for safe operation in robot-assisted surgery. Although robotic forceps that senses the force by force sensors or driving torque of electric motors is proposed, the force sensors and the motors have some problems such as increase in weight and difficulty of the sterilization.

METHOD

We developed a pneumatically driven robotic forceps that estimates the grasping torque and display the magnified torque for the operator. The robotic forceps has a master device and a slave robot, and they are integrated. In the slave side, the grasping torque is estimated by the pressure change in the pneumatic cylinder. A pneumatic bellows display the torque through a linkage.

RESULTS

We confirmed that the slave robot follows the motion of the master, and the grasping torque is estimated in the accuracy of 7 mNm and is magnified and displayed for the operator.

CONCLUSIONS

The pneumatically driven robotic forceps has the capability in the estimation of the grasping torque and display of the torque. Regarding future work, the usability and fatigues of the surgeons must be evaluated.

PHTNet: Characterization and Deep Mining of Involuntary Pathological Hand Tremor using Recurrent Neural Network Models

The global aging phenomenon has increased the number of individuals with age-related neurological movement disorders including Parkinson's Disease (PD) and Essential Tremor (ET). Pathological Hand Tremor (PHT), which is considered among the most common motor symptoms of such disorders, can severely affect patients' independence and quality of life. To develop advanced rehabilitation and assistive technologies, accurate estimation/prediction of nonstationary PHT is critical, however, the required level of accuracy has not yet been achieved. The lack of sizable datasets and generalizable modeling techniques that can fully represent the spectrotemporal characteristics of PHT have been a critical bottleneck in attaining this goal. This paper addresses this unmet need through establishing a deep recurrent model to predict and eliminate the PHT component of hand motion. More specifically, we propose a machine learning-based, assumption-free, and real-time PHT elimination framework, the PHTNet, by incorporating deep bidirectional recurrent neural networks. The PHTNet is developed over a hand motion dataset of 81 ET and PD patients collected systematically in a movement disorders clinic over 3 years. The PHTNet is the first intelligent systems model developed on this scale for PHT elimination that maximizes the resolution of estimation and allows for prediction of future and upcoming sub-movements.

Multi-Modal Haptic Feedback for Grip Force Reduction in Robotic Surgery

Minimally invasive robotic surgery allows for many advantages over traditional surgical procedures, but the loss of force feedback combined with a potential for strong grasping forces can result in excessive tissue damage. Single modality haptic feedback systems have been designed and tested in an attempt to diminish grasping forces, but the results still fall short of natural performance. A multi-modal pneumatic feedback system was designed to allow for tactile, kinesthetic, and vibrotactile feedback, with the aims of more closely imitating natural touch and further improving the effectiveness of HFS in robotic surgical applications and tasks such as tissue grasping and manipulation. Testing of the multi-modal system yielded very promising results with an average force reduction of nearly 50% between the no feedback and hybrid (tactile and kinesthetic) trials (p < 1.0E-16). The multi-modal system demonstrated an increased reduction over single modality feedback solutions and indicated that the system can help users achieve average grip forces closer to those normally possible with the human hand.

Design and Implementation of Respiration Rate Measurement System Using an Information Filter on an Embedded Device

In this work, an algorithm was developed to measure respiration rate for an embedded device that can be used by a field robot for relief operations. With this algorithm, the rate measurement was calculated based on direct influences of respiratory-induced intensity variation (RIIV) on blood flow in cardiovascular pathways. For this, a photoplethysmogram (PPG) sensor was used to determine changes in heartbeat frequencies. The PPG sensor readings were filtered using an Information Filter and a fast Fourier transform (FFT) to determine the state of RIIV. With a relatively light initialization, the information filter can estimate unknown variables based on a series of measurements containing noise and other inaccuracies. Therefore, this filter is suitable for application in an embedded device. For faster calculation time in the implementation, the FFT analysis was calculated only for a major peak in frequency domain. Test and measurement of respiration rate was conducted based on the device algorithm and spirometer. Heartbeat measurements were also evaluated by comparing the heartbeat data of the PPG sensor and pulse-oximeter. Based on the test, the implemented algorithm can measure the respiration rate with approximately 80% accuracy compared with the spirometer.

Magnetorheological Fluids Actuated Haptic-Based Teleoperated Catheter Operating System

During conventional catheter endovascular procedures, surgeons needs to adjust the catheter intervention moving direction and velocity according to the direct sensation. Moreover, in the conventional method, both the surgeon and the patient are inevitable exposed to a large amount of, and for a long period of time, X-ray radiation during the surgical procedure. The purpose of this paper is to ensure surgical safety and to protect the surgeon from X-ray radiation during the surgical procedure by adopting a novel haptic-based robot-assisted master-slave system mode. In this paper, a kind of magnetorheological fluids (MR fluids)-based haptic interface has been developed to generate a kind of controllable haptic sensation providing to the catheter operator, and the catheter intervention kinematics parameters measured the motion capture part to control the salve robotic catheter operating system following the master side kinematics. The slave catheter operating the mechanical system has also been designed and manufactured to manipulate the clinical catheter by mimicking the surgeon operating the catheter intervention surgical procedure, which has a 2-DOF (advance, retreat, and rotate) catheter motion characteristic; in addition, the interaction force between the catheter and inner wall of vasculature can be measured by its force sensing unit and the feedback to the master system. The catheter intervention synchronous evaluation experiments between the master and slave system are tested. Also, the advantages of integrating the controllable haptic sensation to the master-slave system experimental evaluations have been done in vitro. The experimental results demonstrated that the proposed haptic-based robot-assisted master-slave system mode can reduce the surgical time and protect the surgeon from X-ray radiation.

Three-dimensional nonlinear force-sensing method based on double microgrippers with E-type vertical elastomer for minimally invasive robotic surgery

This paper presents a new type of forceps that consist of two microgrippers with the capability of direct force sensing, which enables grasping and manipulating forces at the tip of surgical instrument for minimally invasive robotic surgery. For the prototype design of the forceps, a double E-type vertical elastomer with four strain beams is presented, whose force-sensing principle is expounded. Thus, the forceps with the elastomer can be considered a compliant component, which provides tiny displacements that allow large strain, and the overall diameter is 10 mm. The sizes of the elastomer and forceps are successively determined by analyzing the relationship of several parameters and strain. Then, the linearity analysis of strain beams determines the positions to apply gauges for sensing. The two-dimensional force decoupling models for a single microgripper are proposed based on piecewise analytical polynomials of the strain difference and employed to develop a new three-dimensional force nonlinear decoupling algorithm based on double microgrippers, which realizes single-axial grasping and three-axial pulling forces. Finally, the required force-sensing performance of the proposed method is successfully verified in theory using finite-element simulations.

A Multi-Finger Interface with MR Actuators for Haptic Applications

Haptic devices with multi-finger input are highly desirable in providing realistic and natural feelings when interacting with the remote or virtual environment. Compared with the conventional actuators, MR (Magneto-rheological) actuators are preferable options in haptics because of larger passive torque and torque-volume ratios. Among the existing haptic MR actuators, most of them are still bulky and heavy. If they were smaller and lighter, they would become more suitable for haptics. In this paper, a small-scale yet powerful MR actuator was designed to build a multi-finger interface for the 6 DOF haptic device. The compact structure was achieved by adopting the multi-disc configuration. Based on this configuration, the MR actuator can generate the maximum torque of 480 N.mm with dimensions of only 36 mm diameter and 18 mm height. Performance evaluation showed that it can exhibit a relatively high dynamic range and good response characteristics when compared with some other haptic MR actuators. The multi-finger interface is equipped with three MR actuators and can provide up to 8 N passive force to the thumb, index and middle fingers, respectively. An application example was used to demonstrate the effectiveness and potential of this new MR actuator based interface.

Prevalence of haptic feedback in robot-mediated surgery: a systematic review of literature

With the successful uptake and inclusion of robotic systems in minimally invasive surgery and with the increasing application of robotic surgery (RS) in numerous surgical specialities worldwide, there is now a need to develop and enhance the technology further. One such improvement is the implementation and amalgamation of haptic feedback technology into RS which will permit the operating surgeon on the console to receive haptic information on the type of tissue being operated on. The main advantage of using this is to allow the operating surgeon to feel and control the amount of force applied to different tissues during surgery thus minimising the risk of tissue damage due to both the direct and indirect effects of excessive tissue force or tension being applied during RS. We performed a two-rater systematic review to identify the latest developments and potential avenues of improving technology in the application and implementation of haptic feedback technology to the operating surgeon on the console during RS. This review provides a summary of technological enhancements in RS, considering different stages of work, from proof of concept to cadaver tissue testing, surgery in animals, and finally real implementation in surgical practice. We identify that at the time of this review, while there is a unanimous agreement regarding need for haptic and tactile feedback, there are no solutions or products available that address this need. There is a scope and need for new developments in haptic augmentation for robot-mediated surgery with the aim of improving patient care and robotic surgical technology further.

Novel force-sensing system for minimally invasive surgical instruments

Mastering proper force manipulation in minimally invasive surgery can take many years. Improper force control can lead to necrosis, infection, and scarring. This paper describes a novel system to measure, log, and display external forces at the distal end of minimally invasive surgical instruments in real-time. The system, comprising of a Force- Sensing Sleeve, Bluetooth electronics module, and an Android mobile application. A sensorized 5 mm minimally invasive surgical needle holder was evaluated for bending force accuracy, linearity, and repeatability in six directions. The results showed that the system responded linearly to forces at the tool-tip independent of direction with an RMS error of 0.088 N. Repeatability was affected by system noise potentially arising from temperature drift and thermal noise. Future work will include characterization of communication performance for force feedback in surgical training and assessment.

Influence of surgical gloves on haptic perception thresholds

BACKGROUND

Impairment of haptic perception by surgical gloves could reduce requirements on haptic systems for surgery. While grip forces and manipulation capabilities were not impaired in previous studies, no data is available for perception thresholds.

METHODS

Absolute and differential thresholds (20 dB above threshold) of 24 subjects were measured for frequencies of 25 and 250 Hz with a Ψ-method. Effects of wearing a surgical glove, moisture on the contact surface and subject's experience with gloves were incorporated in a full-factorial experimental design.

RESULTS

Absolute thresholds of 12.8 dB and -29.6 dB (means for 25 and 250 Hz, respectively) and differential thresholds of -12.6 dB and -9.5 dB agree with previous studies. A relevant effect of the frequency on absolute thresholds was found. Comparisons of glove- and no-glove-conditions did not reveal a significant mean difference.

CONCLUSIONS

Wearing a single surgical glove does not affect absolute and differential haptic perception thresholds.

Experimental evaluation of magnified haptic feedback for robot-assisted needle insertion and palpation

BACKGROUND

Haptic feedback has been proven to play a key role in enhancing the performance of teleoperated medical procedures. However, due to safety issues, commercially-available medical robots do not currently provide the clinician with haptic feedback.

METHODS

This work presents the experimental evaluation of a teleoperation system for robot-assisted medical procedures able to provide magnified haptic feedback to the clinician. Forces registered at the operating table are magnified and provided to the clinician through a 7-DoF haptic interface. The same interface is also used to control the motion of a 6-DoF slave robotic manipulator. The safety of the system is guaranteed by a time-domain passivity-based control algorithm.

RESULTS

Two experiments were carried out on stiffness discrimination (during palpation and needle insertion) and one experiment on needle guidance.

CONCLUSIONS

Our haptic-enabled teleoperation system improved the performance with respect to direct hand interaction of 80%, 306%, and 27% in stiffness discrimination through palpation, stiffness discrimination during needle insertion, and guidance, respectively.

A novel 4-DOF surgical instrument with modular joints and 6-Axis Force sensing capability

BACKGROUND

It is difficult for surgeons to exert appropriate forces during delicate operations due to lack of force feedback in robot-assisted minimally invasive surgery (RMIS). A 4-DOF surgical grasper with a modular wrist and 6-axis force sensing capability is developed.

METHODS

A grasper integrated with a miniature force and torque sensor based on the Stewart platform is designed, and a cable tension decomposition mechanism is designed to alleviate influence of the cable tension to the sensor. A modularized wrist consisting of four joint units is designed to facilitate integration of the sensor and eliminate coupled motion of the wrist.

RESULTS

Sensing ranges of this instrument are ±10 N and ±160 N mm, and resolutions are 1.2% in radial directions, 5% in axial direction, and 4.2% in rotational directions. An ex vivo experiment shows that this instrument prototype successfully measures the interaction forces.

CONCLUSIONS

A 4-DOF surgical instrument with modular joints and 6-axis force sensing capability is developed. This instrument can be used for force feedback in RMIS. Copyright © 2016 John Wiley & Sons, Ltd.

A force-sensing surgical tool with a proximally located force/torque sensor

BACKGROUND

Robotic surgery has seen a rapid increase in popularity in the last few decades because advantages such as increased accuracy and dexterity can be realized. These systems still lack force-feedback, where such a capability is believed to be beneficial to the surgeon and can improve safety.

METHOD

In this paper a force-feedback enabled surgical robotic system is described in which the developed force-sensing surgical tool is discussed in detail. The developed surgical tool makes use of a proximally located force/torque sensor, which, in contrast to a distally located sensor, requires no miniaturization or sterilizability.

RESULTS

Experimental results are presented, and indicate high force sensing accuracies with errors <0.09 N.

CONCLUSIONS

It is shown that developing a force-sensing surgical tool utilizing a proximally located force/torque sensor is feasible, where a tool outer diameter of 12 mm can be achieved. For future work it is desired to decrease the current tool outer diameter to 10 mm. Copyright © 2016 John Wiley & Sons, Ltd.

Prevailing Trends in Haptic Feedback Simulation for Minimally Invasive Surgery

Background The amount of direct hand-tool-tissue interaction and feedback in minimally invasive surgery varies from being attenuated in laparoscopy to being completely absent in robotic minimally invasive surgery. The role of haptic feedback during surgical skill acquisition and its emphasis in training have been a constant source of controversy. This review discusses the major developments in haptic simulation as they relate to surgical performance and the current research questions that remain unanswered. Search Strategy An in-depth review of the literature was performed using PubMed. Results A total of 198 abstracts were returned based on our search criteria. Three major areas of research were identified, including advancements in 1 of the 4 components of haptic systems, evaluating the effectiveness of haptic integration in simulators, and improvements to haptic feedback in robotic surgery. Conclusions Force feedback is the best method for tissue identification in minimally invasive surgery and haptic feedback provides the greatest benefit to surgical novices in the early stages of their training. New technology has improved our ability to capture, playback and enhance to utility of haptic cues in simulated surgery. Future research should focus on deciphering how haptic training in surgical education can increase performance, safety, and improve training efficiency.

A modular force-controlled robotic instrument for minimally invasive surgery - efficacy for being used in autonomous grasping against a variable pull force

BACKGROUND

Many deficiencies of minimally invasive robotic surgery systems can be eliminated by using automated laparoscopic tools with force measurement and control capability.

METHOD

A fully modular, automated laparoscopic instrument with a proximal force sensory system was designed and fabricated. The efficacy of the instrument was evaluated experimentally when functioning in an autonomous force-controlled grasping scheme.

RESULTS

The designed instrument was shown to work easily with standard laparoscopic tools, with the whole distal part detachable for autoclave sterilization. The root mean squared error (RMSE) of the actual pinch force from the target ramp was 0.318 N; it was 0.402 N for a sinusoidal pull force, which dropped by 21% using a static friction compensation. A secure grasping condition was achieved, in spite of this error, by applying a sufficiently large margin from the slip boundary.

CONCLUSIONS

With a simple and practical design, the instrument enjoys affordability, versatility and autoclave sterilizability for clinical usage, with an acceptable performance for being used in an auto-grasping control scheme. Copyright © 2016 John Wiley & Sons, Ltd.

A Dual-Arm 7-Degrees-of-Freedom Haptics-Enabled Teleoperation Test Bed for Minimally Invasive Surgery

This paper describes a dual-arm teleoperation (master-slave) system which has been developed to explore the effect of haptics in robotics-assisted minimally invasive surgery (RAMIS). This setup is capable of measuring forces in 7 degrees of freedom (DOF) and fully reflecting them to the operator through two 7-DOF haptic interfaces. An application of the test bed is in enabling the evaluation of the effect of replacing haptic feedback by other sensory cues such as visual representation of haptic information (sensory substitution). This paper discusses the design rationale, kinematic analysis and dynamic modeling of the robot manipulators, and the control system developed for the setup. Using the accurate model developed in this paper, a highly transparent haptics-enabled system can be achieved and used in robot-assisted telesurgery. Validation results obtained through experiments are presented and demonstrate the correctness and effectiveness of the developed models. The application of the setup for two RAMIS surgical tasks, a suture manipulation task and a tumor localization task, is described with different haptics modalities available through the developed haptics-enabled system for each application.

Haptic feedback and control of a flexible surgical endoscopic robot

A flexible endoscope could reach the potential surgical site via a single small incision on the patient or even through natural orifices, making it a very promising platform for surgical procedures. However, endoscopic surgery has strict spatial constraints on both tool-channel size and surgical site volume. It is therefore very challenging to deploy and control dexterous robotic instruments to conduct surgical procedures endoscopically. Pioneering endoscopic surgical robots have already been introduced, but the performance is limited by the flexible neck of the robot that passes through the endoscope tool channel. In this article we present a series of new developments to improve the performance of the robot: a force transmission model to address flexibility, elongation study for precise position control, and tissue property modeling for haptic feedback. Validation experiment results are presented for each sector. An integrated control architecture of the robot system is given in the end.

Introduction to haptics for neurosurgeons

Robots are becoming increasingly relevant to neurosurgeons, extending a neurosurgeon's physical capabilities, improving navigation within the surgical landscape when combined with advanced imaging, and propelling the movement toward minimally invasive surgery. Most surgical robots, however, isolate surgeons from the full range of human senses during a procedure. This forces surgeons to rely on vision alone for guidance through the surgical corridor, which limits the capabilities of the system, requires significant operator training, and increases the surgeon's workload. Incorporating haptics into these systems, ie, enabling the surgeon to "feel" forces experienced by the tool tip of the robot, could render these limitations obsolete by making the robot feel more like an extension of the surgeon's own body. Although the use of haptics in neurosurgical robots is still mostly the domain of research, neurosurgeons who keep abreast of this emerging field will be more prepared to take advantage of it as it becomes more prevalent in operating theaters. Thus, this article serves as an introduction to the field of haptics for neurosurgeons. We not only outline the current and future benefits of haptics but also introduce concepts in the fields of robotic technology and computer control. This knowledge will allow readers to be better aware of limitations in the technology that can affect performance and surgical outcomes, and "knowing the right questions to ask" will be invaluable for surgeons who have purchasing power within their departments.

Integration of force reflection with tactile sensing for minimally invasive robotics-assisted tumor localization

Tactile sensing and force reflection have been the subject of considerable research for tumor localization in soft-tissue palpation. The work presented in this paper investigates the relevance of force feedback (presented visually as well as directly) during tactile sensing (presented visually only) for tumor localization using an experimental setup close to one that could be applied for real robotics-assisted minimally invasive surgery. The setup is a teleoperated (master-slave) system facilitated with a state-of-the-art minimally invasive probe with a rigidly mounted tactile sensor at the tip and an externally mounted force sensor at the base of the probe. The objective is to capture the tactile information and measure the interaction forces between the probe and tissue during palpation and to explore how they can be integrated to improve the performance of tumor localization. To quantitatively explore the effect of force feedback on tactile sensing tumor localization, several experiments were conducted by human subjects to locate artificial tumors embedded in the ex vivo bovine livers. The results show that using tactile sensing in a force-controlled environment can realize, on average, 57 percent decrease in the maximum force and 55 percent decrease in the average force applied to tissue while increasing the tumor detection accuracy by up to 50 percent compared to the case of using tactile feedback alone. The results also show that while visual presentation of force feedback gives straightforward quantitative measures, improved performance of tactile sensing tumor localization is achieved at the expense of longer times for the user. Also, the quickness and intuitive data mapping of direct force feedback makes it more appealing to experienced users.

Magnetic resonance imaging-compatible tactile sensing device based on a piezoelectric array

Minimally invasive surgery is a widely used medical technique, one of the drawbacks of which is the loss of direct sense of touch during the operation. Palpation is the use of fingertips to explore and make fast assessments of tissue morphology. Although technologies are developed to equip minimally invasive surgery tools with haptic feedback capabilities, the majority focus on tissue stiffness profiling and tool-tissue interaction force measurement. For greatly increased diagnostic capability, a magnetic resonance imaging-compatible tactile sensor design is proposed, which allows minimally invasive surgery to be performed under image guidance, combining the strong capability of magnetic resonance imaging soft tissue and intuitive palpation. The sensing unit is based on a piezoelectric sensor methodology, which conforms to the stringent mechanical and electrical design requirements imposed by the magnetic resonance environment The sensor mechanical design and the device integration to a 0.2 Tesla open magnetic resonance imaging scanner are described, together with the device's magnetic resonance compatibility testing. Its design limitations and potential future improvements are also discussed. A tactile sensing unit based on a piezoelectric sensor principle is proposed, which is designed for magnetic resonance imaging guided interventions.

A triple-jaw actuated and sensorized instrument for grasping large organs during minimally invasive robotic surgery

BACKGROUND

Secure grasping and effective manipulation of delicate large organs during robotic surgery operations needs especially designed instruments that can enclose a large amount of tissue and feed back the pinch forces.

METHODS

A large organ triple-jaw grasper was instrumented using practical force sensory and actuating systems. A force tracking scheme was proposed to facilitate auto-grasping of large organs during robotic teleoperation surgery. An on-site force commanding/reflecting mechanism was also implemented to use the device as an independent hand-held robotic instrument. The efficacy of the robotic grasper was examined in phantom tests.

RESULTS

The instrument grasped large soft objects effectively and safely with accurately measured and controlled pinch forces. Furthermore, it could characterize the overall mechanical behavior of the grasping objects.

CONCLUSIONS

The instrument designed provides a potential solution for the safe and effective grasping and manipulation of large abdominal organs, either as a hand-held device, or in a teleoperation framework.

Algorithms for autonomous exploration and estimation in compliant environments

This paper investigates algorithms for enabling surgical slave robots to autonomously explore shape and stiffness of surgical fields. The paper addresses methods for estimating shape and impedance parameters of tissue and methods for autonomously exploring perceived impedance during tool interaction inside a tissue cleft. A hybrid force-motion controller and a cycloidal motion path are proposed to address shape exploration. An adaptive exploration algorithm for segmentation of surface features and a predictor-corrector algorithm for exploration of deep features are introduced based on discrete impedance estimates. These estimates are derived from localized excitation of tissue coupled with simultaneous force measurements. Shape estimation is validated in ex-vivo bovine tissue and attains surface estimation errors of less than 2.5 mm with force sensing resolutions achievable with current technologies in minimally invasive surgical robots. The effect of scan patterns on the accuracy of the shape estimate is demonstrated by comparing the shape estimate of a Cartesian raster scan with overlapping cycloid scan pattern. It is shown that the latter pattern filters the shape estimation bias due to frictional drag forces. Surface impedance exploration is validated to successfully segment compliant environments on flexible inorganic models. Simulations and experiments show that the adaptive search algorithm reduces overall time requirements relative to the complexity of the underlying structures. Finally, autonomous exploration of deep features is demonstrated in an inorganic model and ex-vivo bovine tissue. It is shown that estimates of least constraint based on singular value decomposition of locally estimated tissue stiffness can generate motion to accurately follow a tissue cleft with a predictor-corrector algorithm employing alternating steps of position and admittance control. We believe that these results demonstrate the potential of these algorithms for enabling “smart” surgical devices capable of autonomous execution of intraoperative surgical plans.

Telerobotic-assisted bone-drilling system using bilateral control with feed operation scaling and cutting force scaling

Background

Drilling is used in the medical field, especially in oral surgery and orthopaedics. In recent years, oral surgery involving dental implants has become more common. However, the risky drilling process causes serious accidents. To prevent these accidents, supporting systems such as robotic drilling systems are required.

Methods

A telerobotic-assisted drilling system is proposed. An acceleration-based four-channel bilateral control system is implemented in linear actuators in a master–slave system for drill feeding. A reaction force observer is used instead of a force sensor for measuring cutting force. Cutting force transmits from a cutting material to a surgeon, who may feel a static cutting resistance force and vigorous cutting vibrations, via the master–slave system. Moreover, position scaling and force scaling are achieved. Scaling functions are used to achieve precise drilling and hazard detection via force sensation.

Results

Cutting accuracy and reproducibility of the cutting force were evaluated by angular velocity/position error and frequency analysis of the cutting force, respectively, and errors were > 2.0 rpm and > 0.2 mm, respectively. Spectrum peaks of the cutting vibration were at the theoretical vibration frequencies of 30, 60 and 90 Hz.

Conclusions

The proposed telerobotic-assisted drilling system achieved precise manipulation of the drill feed and vivid feedback from the cutting force. Copyright © 2012 John Wiley & Sons, Ltd.

Advances in Haptics, Tactile Sensing, and Manipulation for Robot-Assisted Minimally Invasive Surgery, Noninvasive Surgery, and Diagnosis

The developments of medical practices and medical technologies have always progressed concurrently. The relatively recent developments in endoscopic technologies have allowed the realization of the “minimally invasive” form of surgeries. The advancements in robotics facilitate precise surgeries that are often integrated with medical image guidance capability. This in turn has driven the further development of technology to compensate for the unique complexities engendered by this new format and to improve the performance and broaden the scope of the procedures that can be performed. Medical robotics has been a central component of this development due to the highly suitable characteristics that a robotic system can purport, including highly optimizable mechanical conformation and the ability to program assistive functions in medical robots for surgeons to perform safe and accurate minimally invasive surgeries. In addition, combining the robot-assisted interventions with touch-sensing and medical imaging technologies can greatly improve the available information and thus help to ensure that minimally invasive surgeries continue to gain popularity and stay at the focus of modern medical technology development. This paper presents a state-of-the-art review of robotic systems for minimally invasive and noninvasive surgeries, precise surgeries, diagnoses, and their corresponding technologies.

Medical Robots: Current Systems and Research Directions

First used medically in 1985, robots now make an impact in laparoscopy, neurosurgery, orthopedic surgery, emergency response, and various other medical disciplines. This paper provides a review of medical robot history and surveys the capabilities of current medical robot systems, primarily focusing on commercially available systems while covering a few prominent research projects. By examining robotic systems across time and disciplines, trends are discernible that imply future capabilities of medical robots, for example, increased usage of intraoperative images, improved robot arm design, and haptic feedback to guide the surgeon.

Photonic crystal fiber sensors for minimally invasive surgical devices

The measurement of interaction forces in minimally invasive surgical devices, sensorized with photonic crystal fiber (PCF) sensors, is presented in this paper. Two types of PCF sensors are used: a tapered PCF interferometer and a microhole-collapsed PCF interferometer for the detection of interaction forces generated in surgical devices without the influence of ambient temperature variation. The demonstration devices used for force characterization are a laparoscopic scissor and a standard surgical scissor blade. The force sensitivity of each sensorized blade is examined and compared with fiber Bragg grating (FBG)-sensorized blades. Results show that the PCF-sensorized surgical blades outperform the blades fitted with the FBG sensors during static load measurement.

Micron: an Actively Stabilized Handheld Tool for Microsurgery

We describe the design and performance of a hand-held actively stabilized tool to increase accuracy in micro-surgery or other precision manipulation. It removes involuntary motion such as tremor by actuating the tip to counteract the effect of the undesired handle motion. The key components are a three-degree-of-freedom piezoelectric manipulator that has 400 μm range of motion, 1 N force capability, and bandwidth over 100 Hz, and an optical position measurement subsystem that acquires the tool pose with 4 μm resolution at 2000 samples/s. A control system using these components attenuates hand motion by at least 15 dB (a fivefold reduction). By considering the effect of the frequency response of Micron on the human visual feedback loop, we have developed a filter that reduces unintentional motion, yet preserves intuitive eye-hand coordination. We evaluated the effectiveness of Micron by measuring the accuracy of the human/machine system in three simple manipulation tasks. Handheld testing by three eye surgeons and three non-surgeons showed a reduction in position error of between 32% and 52%, depending on the error metric.

Object manipulation improvements due to single session training outweigh the differences among stimulation sites during vibrotactile feedback

Most hand prostheses do not provide intentional haptic feedback about movement performance; thus users must rely almost completely on visual feedback. This paper focuses on understanding the effects of learning and different stimulation sites when vibrotactile stimulation is used as the intentional haptic feedback. Eighteen unimpaired individuals participated in this study with a robotic interface to manipulate a virtual object with visual and vibrotactile feedback at four body sites (finger, arm, neck, and foot) presented in a random order. All participants showed improvements in object manipulation performance with the addition of vibrotactile feedback. Specifically, performance showed a strong learning effect across time, with learning transferring across different sites of vibrotactile stimulation. The effects of learning over the experiment overshadowed the effects of different stimulation sites. The addition of a cognitive task slowed participants and increased the subjective difficulty. User preference ratings showed no difference in their preference among vibrotactile stimulation sites. These findings indicate that the stimulation site may not be as critical as ensuring adequate training with vibrotactile feedback during object manipulation. Future research to identify improvements in vibrotactile-based feedback parameters with amputees is warranted.

Tool Contact Acceleration Feedback for Telerobotic Surgery

Minimally invasive telerobotic surgical systems enable surgeons to perform complicated procedures without large incisions. Unfortunately, these systems typically do not provide the surgeon with sensory feedback aside from stereoscopic vision. We have, thus, developed VerroTouch, a sensing and actuating device that can be added to Intuitive Surgical's existing da Vinci S Surgical System to provide auditory and vibrotactile feedback of tool contact accelerations. These cues let the surgeon feel and hear contact with rough textures as well as the making and breaking of contact with objects and other tools. To evaluate the merits of this approach, we had 11 surgeons use an augmented da Vinci S to perform three in vitro manipulation tasks under four different feedback conditions: with no acceleration feedback, with audio feedback, with haptic feedback, and with both audio and haptic. Subjects expressed a significant preference for the inclusion of tool contact acceleration feedback, although they disagreed over which sensory modality was best. Other survey responses and qualitative written comments indicate that the feedback may have improved the subject's concentration and situational awareness by strengthening the connection between the surgeon and the surgical instruments. Analysis of quantitative task metrics shows that the feedback neither improves nor impedes the performance of the chosen tasks.

Robotic endoscope motor module and gearing design

Actuation of a robotic endoscope with increased torque output is presented. This paper will specifically focus on the motor module section of a robotic endoscope, which comprises of a pair of motors and gear reduction assemblies. The results for the endoscope and biopsy tool stiffness, as well as the stall force and force versus speed characteristics of the motor module assembly are shown. The scope stiffness was found to be 0.006 N/degree and additional stiffness of the biopsy tools were found to be in the range of 0.09 to 0.13 N/degree. Calculations for worm gearing and efficiency are discussed.

Design and fabrication of a novel tactile sensory system applicable in artificial palpation

Force and position feedback are the two important parameters that are employed in different medical diagnoses and more specifically surgical operations. Furthermore, during different minimally invasive procedures, the ability of touch and force and position feedback are absent. In this regard, artificial palpation is a new technology that is employed to obtain tactile data in situations where physicians/surgeons cannot use their tactile sense. One of the most valuable achievements of artificial palpation are tactile sensory systems that have various applications in the detection of hard objects inside the soft tissue. Considering the present problems and limitations of kidney stone removal laparoscopy, the aim of this research is to design and fabricate a novel tactile sensory system capable of determining the exact location of stones during laparoscopy. This new tactile sensory system consists of four main parts: The sensory part, the mechanical part, the electrical part, and the display part. In this new system, due to the use of both displacement and force sensors, the usage limitations of previous tactile sensory systems are eliminated. The new tactile sensory system is well capable of finding the stone in the laboratory models through physical contact with the model's surface.

Tendon sheath analysis for estimation of distal end force and elongation for sensorless distal end

Tendon sheath actuation is found in many applications, particularly in robotic hands and surgical robots. Due to the friction between the tendon and sheath, many undesirable characteristic such as backlash, hysteresis and non-linearity are present. It is desirable to know the end-effector force and elongation of the tendon to control the system effectively, but it is not always feasible to fix sensors at the end effector. A method to estimate the end-effector parameters using only a force and position sensor at the proximal site is given. An analytical study is presented and experiments are reported to support the result, showing a maximum full-scale error of approximately 7%. This result is achieved if the shape of the sheath remains the same and buckling is negligible. The results presented in this study could contribute towards haptic development in robotics surgery.

Disposable endoscope tip actuation design and robotic platform

A disposable endoscopic platform with actuation motors inside the body of the endoscope is presented. This platform can enable new medical devices for diagnosis and for minimally invasive surgeries. This paper addresses mechanical and safety issues with existing endoscope technologies by incorporating disposability, safety modules, and lower cable forces. In order to produce path-independent cable forces, motors are incorporated in the body of the endoscope near the bending tip. Results for tip forces are shown accompanied by an analytical model describing the scaling laws for this type of robotic architecture. The system under development will provide a platform for research into haptic control and perceptual feedback.

A Sensorized Instrument for Skills Assessment and Training in Minimally Invasive Surgery

Minimally invasive surgery (MIS) is carried out using long, narrow instruments and significantly reduces trauma to the body, postoperative pain, and recovery time. Unfortunately, the restricted access conditions, limited instrument motion, and degraded sense of touch inherent in MIS result in new perceptual-motor relationships, which are unfamiliar to the surgeon and require training to overcome. Current training methods do not adequately address the needs of surgeons interested in acquiring these skills. Although a significant amount of research has been focused on the development of sensorized systems for surgery, there is still a need for a system that can be used in any training scenario (laparoscopic trainer, animal laboratories, or real surgery) for the purpose of skills assessment and training. A sensorized laparoscopic instrument has been designed that is capable of noninvasively measuring its interaction with tissue in the form of forces or torques acting in all five degrees-of-freedom (DOFs) available during MIS. Strain gauges attached to concentric shafts within the instrument allow the forces acting in different directions to be isolated. An electromagnetic tracking system is used for position tracking. Two prototypes of the sensorized instrument were constructed. Position calibration shows a maximum root mean square (RMS) error of 1.3 mm. The results of the force calibration show a maximum RMS error of 0.35 N for the actuation force, 0.07 N in the x and y directions, and 1.5 N mm for the torque calibration with good repeatability and low hysteresis. Axial measurements were significantly affected by drift, noise, and coupling leading to high errors in the readings. Novel sensorized instruments for skills assessment and training have been developed and a patent has been filed for the design and operation. The instruments measure forces and torques acting at the tip of the instrument corresponding to all five DOFs available during MIS and provide position feedback in six DOFs. The instruments are similar in shape, size, and weight to traditional laparoscopic instruments allowing them to be used in any training environment. Furthermore, replaceable tips and handles allow the instruments to be used for a variety of different tasks.

The value of haptic feedback in conventional and robot-assisted minimal invasive surgery and virtual reality training: a current review

BACKGROUND

Virtual reality (VR) as surgical training tool has become a state-of-the-art technique in training and teaching skills for minimally invasive surgery (MIS). Although intuitively appealing, the true benefits of haptic (VR training) platforms are unknown. Many questions about haptic feedback in the different areas of surgical skills (training) need to be answered before adding costly haptic feedback in VR simulation for MIS training. This study was designed to review the current status and value of haptic feedback in conventional and robot-assisted MIS and training by using virtual reality simulation.

METHODS

A systematic review of the literature was undertaken using PubMed and MEDLINE. The following search terms were used: Haptic feedback OR Haptics OR Force feedback AND/OR Minimal Invasive Surgery AND/OR Minimal Access Surgery AND/OR Robotics AND/OR Robotic Surgery AND/OR Endoscopic Surgery AND/OR Virtual Reality AND/OR Simulation OR Surgical Training/Education.

RESULTS

The results were assessed according to level of evidence as reflected by the Oxford Centre of Evidence-based Medicine Levels of Evidence.

CONCLUSIONS

In the current literature, no firm consensus exists on the importance of haptic feedback in performing minimally invasive surgery. Although the majority of the results show positive assessment of the benefits of force feedback, results are ambivalent and not unanimous on the subject. Benefits are least disputed when related to surgery using robotics, because there is no haptic feedback in currently used robotics. The addition of haptics is believed to reduce surgical errors resulting from a lack of it, especially in knot tying. Little research has been performed in the area of robot-assisted endoscopic surgical training, but results seem promising. Concerning VR training, results indicate that haptic feedback is important during the early phase of psychomotor skill acquisition.

Robot-assisted Tactile Sensing for Minimally Invasive Tumor Localization

The 10 mm incisions used in minimally invasive cancer surgery prevent the direct palpation of internal organs, making intraoperative tumor localization difficult. A tactile sensing instrument (TSI), which uses a commercially available sensor to measure distributed pressure profiles along the contacting surface, has been developed to facilitate remote tissue palpation. The objective of this research is to assess the feasibility of using the TSI under robotic control to reliably locate underlying tumors while reducing collateral tissue trauma. The performance of humans and a robot using the TSI to locate tumor phantoms embedded into ex vivo bovine livers is compared. An augmented hybrid impedance control scheme has been implemented on a Mitsubishi PA10-7C to perform the force/position control used in the trials. The results show that using the TSI under robotic control realizes an average 35% decrease in the maximum forces applied and a 50% increase in tumor detection accuracy when compared to manual manipulation of the same instrument. This demonstrates that the detection of tumors using tactile sensing is highly dependent on how consistently the forces on the tactile sensing area are applied, and that robotic assistance can be of great benefit when trying to localize tumors in minimally invasive surgery.

A Multielement Tactile Feedback System for Robot-Assisted Minimally Invasive Surgery

A multi-element tactile feedback (MTF) system has been developed to translate the force distribution, in magnitude and position, from 3times2 sensor arrays on surgical robotic end-effectors to the fingers via 3times2 balloon tactile displays. High detection accuracies from perceptual tests (> 96%) suggest that MTF may be an effective means to improve robotic control.