1
|
Oquendo YA, Coad MM, Wren SM, Lendvay TS, Nisky I, Jarc AM, Okamura AM, Chua Z. Haptic Guidance and Haptic Error Amplification in a Virtual Surgical Robotic Training Environment. IEEE TRANSACTIONS ON HAPTICS 2024; 17:417-428. [PMID: 38194379 DOI: 10.1109/toh.2024.3350128] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/11/2024]
Abstract
Teleoperated robotic systems have introduced more intuitive control for minimally invasive surgery, but the optimal method for training remains unknown. Recent motor learning studies have demonstrated that exaggeration of errors helps trainees learn to perform tasks with greater speed and accuracy. We hypothesized that training in a force field that pushes the user away from a desired path would improve their performance on a virtual reality ring-on-wire task. Thirty-eight surgical novices trained under a no-force, guidance, or error-amplifying force field over five days. Completion time, translational and rotational path error, and combined error-time were evaluated under no force field on the final day. The groups significantly differed in combined error-time, with the guidance group performing the worst. Error-amplifying field participants did not plateau in their performance during training, suggesting that learning was still ongoing. Guidance field participants had the worst performance on the final day, confirming the guidance hypothesis. Observed trends also suggested that participants who had high initial path error benefited more from guidance. Error-amplifying and error-reducing haptic training for robot-assisted telesurgery benefits trainees of different abilities differently, with our results indicating that participants with high initial combined error-time benefited more from guidance and error-amplifying force field training.
Collapse
|
2
|
Vandevoorde K, Vollenkemper L, Schwan C, Kohlhase M, Schenck W. Using Artificial Intelligence for Assistance Systems to Bring Motor Learning Principles into Real World Motor Tasks. SENSORS (BASEL, SWITZERLAND) 2022; 22:2481. [PMID: 35408094 PMCID: PMC9002555 DOI: 10.3390/s22072481] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 02/18/2022] [Revised: 03/18/2022] [Accepted: 03/20/2022] [Indexed: 11/03/2022]
Abstract
Humans learn movements naturally, but it takes a lot of time and training to achieve expert performance in motor skills. In this review, we show how modern technologies can support people in learning new motor skills. First, we introduce important concepts in motor control, motor learning and motor skill learning. We also give an overview about the rapid expansion of machine learning algorithms and sensor technologies for human motion analysis. The integration between motor learning principles, machine learning algorithms and recent sensor technologies has the potential to develop AI-guided assistance systems for motor skill training. We give our perspective on this integration of different fields to transition from motor learning research in laboratory settings to real world environments and real world motor tasks and propose a stepwise approach to facilitate this transition.
Collapse
Affiliation(s)
- Koenraad Vandevoorde
- Center for Applied Data Science (CfADS), Faculty of Engineering and Mathematics, Bielefeld University of Applied Sciences, 33619 Bielefeld, Germany; (L.V.); (C.S.); (M.K.)
| | | | | | | | - Wolfram Schenck
- Center for Applied Data Science (CfADS), Faculty of Engineering and Mathematics, Bielefeld University of Applied Sciences, 33619 Bielefeld, Germany; (L.V.); (C.S.); (M.K.)
| |
Collapse
|
3
|
Zruya O, Sharon Y, Kossowsky H, Forni F, Geftler A, Nisky I. A New Power Law Linking the Speed to the Geometry of Tool-Tip Orientation in Teleoperation of a Robot-Assisted Surgical System. IEEE Robot Autom Lett 2022. [DOI: 10.1109/lra.2022.3193485] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
Affiliation(s)
- Or Zruya
- Department of Biomedical Engineering and the Zlotowsky Center for Neuroscience, Ben-Gurion University of the Negev, Beer Sheva, Israel
| | - Yarden Sharon
- Department of Biomedical Engineering and the Zlotowsky Center for Neuroscience, Ben-Gurion University of the Negev, Beer Sheva, Israel
| | - Hanna Kossowsky
- Department of Biomedical Engineering and the Zlotowsky Center for Neuroscience, Ben-Gurion University of the Negev, Beer Sheva, Israel
| | - Fulvio Forni
- Department of Engineering, University of Cambridge, Cambridge, U.K
| | - Alex Geftler
- Department of Orthopedic Surgery, Soroka Medical Center, Ben-Gurion University of the Negev, Beer Sheva, Israel
| | - Ilana Nisky
- Department of Biomedical Engineering and the Zlotowsky Center for Neuroscience, Ben-Gurion University of the Negev, Beer Sheva, Israel
| |
Collapse
|
4
|
Dehghani H, Sun Y, Cubrich L, Oleynikov D, Farritor S, Terry B. An Optimization-Based Algorithm for Trajectory Planning of an Under-Actuated Robotic Arm to Perform Autonomous Suturing. IEEE Trans Biomed Eng 2020; 68:1262-1272. [PMID: 32946377 DOI: 10.1109/tbme.2020.3024632] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
Abstract
In single-port access surgeries, robot size is crucial due to the limited workspace. Thus, a robot may be designed under-actuated. Suturing, in contrast, is a complicated task and requires full actuation. This study aims to overcome this shortcoming by implementing an optimization-based algorithm for autonomous suturing for an under-actuated robot. The proposed algorithm approximates the ideal suturing trajectory by slightly reorienting the needle while deviating as little as possible from the ideal, full degree-of-freedom suturing case. The deviation of the path taken by a custom robot with respect to the ideal trajectory varies depending on the suturing starting location within the workspace as well as the needle size. A quantitative analysis reveals that in 13% of the investigated workspace, the accumulative deviation was less than 10 mm. In the remaining workspace, the accumulative deviation was less than 30 mm. Likewise, the accumulative deviation of a needle with a radius of 10 mm was 2.2 mm as opposed to 8 mm when the radius was 20 mm. The optimization-based algorithm maximized the accuracy of a four-DOF robot to perform a path-constrained trajectory and illustrates the accuracy-workspace correlation.
Collapse
|
5
|
Bahar L, Sharon Y, Nisky I. Surgeon-Centered Analysis of Robot-Assisted Needle Driving Under Different Force Feedback Conditions. Front Neurorobot 2020; 13:108. [PMID: 32038218 PMCID: PMC6993204 DOI: 10.3389/fnbot.2019.00108] [Citation(s) in RCA: 11] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/30/2019] [Accepted: 12/06/2019] [Indexed: 11/24/2022] Open
Abstract
Robotic assisted minimally invasive surgery (RAMIS) systems present many advantages to the surgeon and patient over open and standard laparoscopic surgery. However, haptic feedback, which is crucial for the success of many surgical procedures, is still an open challenge in RAMIS. Understanding the way that haptic feedback affects performance and learning can be useful in the development of haptic feedback algorithms and teleoperation control systems. In this study, we examined the performance and learning of inexperienced participants under different haptic feedback conditions in a task of surgical needle driving via a soft homogeneous deformable object-an artificial tissue. We designed an experimental setup to characterize their movement trajectories and the forces that they applied on the artificial tissue. Participants first performed the task in an open condition, with a standard surgical needle holder, followed by teleoperation in one of three feedback conditions: (1) no haptic feedback, (2) haptic feedback based on position exchange, and (3) haptic feedback based on direct recording from a force sensor, and then again with the open needle holder. To quantify the effect of different force feedback conditions on the quality of needle driving, we developed novel metrics that assess the kinematics of needle driving and the tissue interaction forces, and we combined our novel metrics with classical metrics. We analyzed the final teleoperated performance in each condition, the improvement during teleoperation, and the aftereffect of teleoperation on the performance when using the open needle driver. We found that there is no significant difference in the final performance and in the aftereffect between the 3 conditions. Only the two conditions with force feedback presented statistically significant improvement during teleoperation in several of the metrics, but when we compared directly between the improvements in the three different feedback conditions none of the effects reached statistical significance. We discuss possible explanations for the relative similarity in performance. We conclude that we developed several new metrics for the quality of surgical needle driving, but even with these detailed metrics, the advantage of state of the art force feedback methods to tasks that require interaction with homogeneous soft tissue is questionable.
Collapse
Affiliation(s)
| | | | - Ilana Nisky
- Department of Biomedical Engineering, Zlotowski Center of Neuroscience, Ben-Gurion University of the Negev, Be'er Sheva, Israel
| |
Collapse
|
6
|
Bioinspired Implementation and Assessment of a Remote-Controlled Robot. Appl Bionics Biomech 2019; 2019:8575607. [PMID: 31611928 PMCID: PMC6755284 DOI: 10.1155/2019/8575607] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/26/2019] [Revised: 05/09/2019] [Accepted: 08/21/2019] [Indexed: 11/17/2022] Open
Abstract
Daily activities are characterized by an increasing interaction with smart machines that present a certain level of autonomy. However, the intelligence of such electronic devices is not always transparent for the end user. This study is aimed at assessing the quality of the remote control of a mobile robot whether the artefact exhibits a human-like behavior or not. The bioinspired behavior implemented in the robot is the well-described two-thirds power law. The performance of participants who teleoperate the semiautonomous vehicle implementing the biological law is compared to a manual and nonbiological mode of control. The results show that the time required to complete the path and the number of collisions with obstacles are significantly lower in the biological condition than in the two other conditions. Also, the highest percentage of occurrences of curvilinear or smooth trajectories are obtained when the steering is assisted by an integration of the power law in the robot's way of working. This advanced analysis of the performance based on the naturalness of the movement kinematics provides a refined evaluation of the quality of the Human-Machine Interaction (HMI). This finding is consistent with the hypothesis of a relationship between the power law and jerk minimization. In addition, the outcome of this study supports the theory of a CNS origin of the power law. The discussion addresses the implications of the anthropocentric approach to enhance the HMI.
Collapse
|
7
|
Abdelaal AE, Sakr M, Avinash A, Mohammed SK, Bajwa AK, Sahni M, Hor S, Fels S, Salcudean SE. Play Me Back: A Unified Training Platform for Robotic and Laparoscopic Surgery. IEEE Robot Autom Lett 2019. [DOI: 10.1109/lra.2018.2890209] [Citation(s) in RCA: 11] [Impact Index Per Article: 2.2] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
|
8
|
Shahbazi M, Atashzar SF, Ward C, Talebi HA, Patel RV. Multimodal Sensorimotor Integration for Expert-in-the-Loop Telerobotic Surgical Training. IEEE T ROBOT 2018. [DOI: 10.1109/tro.2018.2861916] [Citation(s) in RCA: 15] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
|
9
|
Basov S, Milstein A, Sulimani E, Platkov M, Peretz E, Rattunde M, Wagner J, Netz U, Katzir A, Nisky I. Robot-assisted laser tissue soldering system. BIOMEDICAL OPTICS EXPRESS 2018; 9:5635-5644. [PMID: 30460151 PMCID: PMC6238920 DOI: 10.1364/boe.9.005635] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/16/2018] [Revised: 09/23/2018] [Accepted: 09/23/2018] [Indexed: 05/30/2023]
Abstract
Fast and reliable incision closure is critical in any surgical intervention. Common solutions are sutures and clips or adhesives, but they all present difficulties. These difficulties are especially pronounced in classical and robot-assisted minimally-invasive interventions. Laser soldering methods present a promising alternative, but their reproducibility is limited. We present a system that combines a previously reported laser soldering system with a robotic system, and demonstrate its feasibility on the incision-closure of ex-vivo mice skins. In this demonstration, we measured tearing forces of ~2.5N, 73% of the tearing force of a mouse skin without an incision. This robot-assisted laser soldering technique has the potential to make laser tissue soldering more reproducible and revolutionize surgical tissue bonding.
Collapse
Affiliation(s)
- Svetlana Basov
- Department of Biomedical Engineering, Tel Aviv University, Tel Aviv, 6997801, Israel
| | - Amit Milstein
- Department of Biomedical Engineering, Ben-Gurion University of the Negev, Beer-Sheva, 8410501, Israel
| | - Erez Sulimani
- Department of Biomedical Engineering, Ben-Gurion University of the Negev, Beer-Sheva, 8410501, Israel
| | - Max Platkov
- Nuclear Research Center Negev, Beer-Sheva, 84190, Israel
| | - Eli Peretz
- Department of Biomedical Engineering, Ben-Gurion University of the Negev, Beer-Sheva, 8410501, Israel
| | - Marcel Rattunde
- Fraunhofer-Institut für Angewandte Festkörperphysik, Freiburg, Germany
| | - Joachim Wagner
- Fraunhofer-Institut für Angewandte Festkörperphysik, Freiburg, Germany
| | - Uri Netz
- Department of Surgery A, Soroka University Medical Center, Beer-Sheva, 85025, Israel
| | - Abraham Katzir
- School of Physics & Astronomy, Tel Aviv University, Tel Aviv, 6997801, Israel
| | - Ilana Nisky
- Department of Biomedical Engineering, Ben-Gurion University of the Negev, Beer-Sheva, 8410501, Israel
| |
Collapse
|
10
|
Leib R, Rubin I, Nisky I. Force feedback delay affects perception of stiffness but not action, and the effect depends on the hand used but not on the handedness. J Neurophysiol 2018; 120:781-794. [PMID: 29766763 DOI: 10.1152/jn.00822.2017] [Citation(s) in RCA: 11] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/22/2022] Open
Abstract
Interaction with an object often requires the estimation of its mechanical properties. We examined whether the hand that is used to interact with the object and their handedness affected people's estimation of these properties using stiffness estimation as a test case. We recorded participants' responses on a stiffness discrimination of a virtual elastic force field and the grip force applied on the robotic device during the interaction. In half of the trials, the robotic device delayed the participants' force feedback. Consistent with previous studies, delayed force feedback biased the perceived stiffness of the force field. Interestingly, in both left-handed and right-handed participants, for the delayed force field, there was even less perceived stiffness when participants used their left hand than their right hand. This result supports the idea that haptic processing is affected by laterality in the brain, not by handedness. Consistent with previous studies, participants adjusted their applied grip force according to the correct size and timing of the load force regardless of the hand that was used, the handedness, or the delay. This suggests that in all of these conditions, participants were able to form an accurate internal representation of the anticipated trajectory of the load force (size and timing) and that this representation was used for accurate control of grip force independently of the perceptual bias. Thus these results provide additional evidence for the dissociation between action and perception in the processing of delayed information. NEW & NOTEWORTHY Introducing delay to force feedback during interaction with an elastic force field biases the perceived stiffness of the force field. We show that this bias depends on the hand that was used for probing but not on handedness. At the same time, both left-handed and right-handed participants adjusted their applied grip force while using either their left or right hands in anticipation of the correct magnitude and timing despite the delay in load force.
Collapse
Affiliation(s)
- Raz Leib
- Department of Biomedical Engineering, Ben-Gurion University of the Negev, Beersheba, Israel.,Zlotowski Center for Neuroscience, Ben-Gurion University of the Negev, Beersheba, Israel
| | - Inbar Rubin
- Department of Biomedical Engineering, Ben-Gurion University of the Negev, Beersheba, Israel
| | - Ilana Nisky
- Department of Biomedical Engineering, Ben-Gurion University of the Negev, Beersheba, Israel.,Zlotowski Center for Neuroscience, Ben-Gurion University of the Negev, Beersheba, Israel
| |
Collapse
|
11
|
Pacchierotti C, Ongaro F, van den Brink F, Yoon C, Prattichizzo D, Gracias DH, Misra S. Steering and control of miniaturized untethered soft magnetic grippers with haptic assistance. IEEE TRANSACTIONS ON AUTOMATION SCIENCE AND ENGINEERING : A PUBLICATION OF THE IEEE ROBOTICS AND AUTOMATION SOCIETY 2018; 15:290-306. [PMID: 31423113 PMCID: PMC6697175 DOI: 10.1109/tase.2016.2635106] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/02/2023]
Abstract
Untethered miniature robotics have recently shown promising results in several scenarios at the microscale, such as targeted drug delivery, microassembly, and biopsy procedures. However, the vast majority of these small-scale robots have very limited manipulation capabilities, and none of the steering systems currently available enable humans to intuitively and effectively control dexterous miniaturized robots in a remote environment. In this paper, we present an innovative micro teleoperation system with haptic assistance for the intuitive steering and control of miniaturized self-folding soft magnetic grippers in 2-D space. The soft grippers can be wirelessly positioned using weak magnetic fields and opened/closed by changing their temperature. An image-guided algorithm tracks the position of the controlled miniaturized gripper in the remote environment. A haptic interface provides the human operator with compelling haptic sensations about the interaction between the gripper and the environment, as well as enables the operator to intuitively control the target position and grasping configuration of the gripper. Finally, magnetic and thermal control systems regulate the position and grasping configuration of the gripper. The viability of the proposed approach is demonstrated through two experiments involving 26 human subjects. Providing haptic stimuli elicited statistically significant improvements in the performance of the considered navigation and micromanipulation tasks. Note to Practitioners-The ability to accurately and intuitively control the motion of miniaturized grippers in remote environments can open new exciting possibilities in the fields of minimally-invasive surgery, micromanipulation, biopsy, and drug delivery. This paper presents a micro teleoperation system with haptic assistance through which a clinician can easily control the motion and open/close capability of miniaturized wireless soft grippers. It introduces the underlying autonomous magnetic and thermal control systems, their interconnection with the master haptic interface, and an extensive evaluation in two real-world scenarios: following of a predetermined trajectory, and pick-and-place of a microscopic object.
Collapse
Affiliation(s)
- C. Pacchierotti
- CNRS at Irisa and Inria Rennes Bretagne Atlantique, Campus de Beaulieu, 35042 Rennes Cedex, France
| | - F. Ongaro
- Surgical Robotics Laboratory, Department of Biomechanical Engineering, MIRA–Institute for Biomedical Technology and Technical Medicine, University of Twente, 7522 NB Enschede, The Netherlands
| | - F. van den Brink
- Surgical Robotics Laboratory, Department of Biomechanical Engineering, MIRA–Institute for Biomedical Technology and Technical Medicine, University of Twente, 7522 NB Enschede, The Netherlands
| | - C. Yoon
- Department of Materials Science and Engineering, The Johns Hopkins University, Baltimore, MD 21218 USA
| | - D. Prattichizzo
- Department of Information Engineering and Mathematics, University of Siena, 53100 Siena, Italy, and also with the Department of Advanced Robotics, Istituto Italiano di Tecnologia, 16163 Genoa, Italy
| | - D. H. Gracias
- Department of Materials Science and Engineering, The Johns Hopkins University, Baltimore, MD 21218 USA
- Department of Chemical and Biomolecular Engineering, The Johns Hopkins University, Baltimore, MD 21218 USA
| | - S. Misra
- Surgical Robotics Laboratory, Department of Biomechanical Engineering, MIRA–Institute for Biomedical Technology and Technical Medicine, University of Twente, 7522 NB Enschede, The Netherlands
- Department of Biomedical Engineering, University Medical Centre Groningen, University of Groningen, 9713 GZ Groningen, The Netherlands
| |
Collapse
|
12
|
Jarc AM, Curet MJ. Viewpoint matters: objective performance metrics for surgeon endoscope control during robot-assisted surgery. Surg Endosc 2016; 31:1192-1202. [PMID: 27422247 PMCID: PMC5315708 DOI: 10.1007/s00464-016-5090-8] [Citation(s) in RCA: 16] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/13/2016] [Accepted: 07/05/2016] [Indexed: 12/16/2022]
Abstract
Background Effective visualization of the operative field is vital to surgical safety and education. However, additional metrics for visualization are needed to complement other common measures of surgeon proficiency, such as time or errors. Unlike other surgical modalities, robot-assisted minimally invasive surgery (RAMIS) enables data-driven feedback to trainees through measurement of camera adjustments. The purpose of this study was to validate and quantify the importance of novel camera metrics during RAMIS. Methods New (n = 18), intermediate (n = 8), and experienced (n = 13) surgeons completed 25 virtual reality simulation exercises on the da Vinci Surgical System. Three camera metrics were computed for all exercises and compared to conventional efficiency measures. Results Both camera metrics and efficiency metrics showed construct validity (p < 0.05) across most exercises (camera movement frequency 23/25, camera movement duration 22/25, camera movement interval 19/25, overall score 24/25, completion time 25/25). Camera metrics differentiated new and experienced surgeons across all tasks as well as efficiency metrics. Finally, camera metrics significantly (p < 0.05) correlated with completion time (camera movement frequency 21/25, camera movement duration 21/25, camera movement interval 20/25) and overall score (camera movement frequency 20/25, camera movement duration 19/25, camera movement interval 20/25) for most exercises. Conclusions We demonstrate construct validity of novel camera metrics and correlation between camera metrics and efficiency metrics across many simulation exercises. We believe camera metrics could be used to improve RAMIS proficiency-based curricula.
Collapse
Affiliation(s)
- Anthony M Jarc
- Medical Research, Intuitive Surgical, Inc., 5655 Spalding Drive, Norcross, GA, 30092, USA.
| | - Myriam J Curet
- Medical Research, Intuitive Surgical, Inc., 5655 Spalding Drive, Norcross, GA, 30092, USA
- VA Palo Alto, Stanford, CA, USA
| |
Collapse
|
13
|
Gibo TL, Abbink DA. Movement Strategy Discovery during Training via Haptic Guidance. IEEE TRANSACTIONS ON HAPTICS 2016; 9:243-254. [PMID: 26766379 DOI: 10.1109/toh.2016.2516984] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/05/2023]
Abstract
Haptic guidance has previously been investigated to facilitate motor skill training, whereupon a robotic device assists a trainee in executing the desired movement. However, many studies have reported a null or even detrimental effect of haptic guidance on training compared to unassisted practice. While prior studies have focused on using haptic guidance to refine a movement strategy, our study explores its effect on the discovery of a new strategy. Subjects learned to manipulate a virtual under-actuated system via a haptic device either with or without haptic guidance (and without haptic feedback of system dynamics). The guidance enabled subjects to experience a range of successful movements, rather than strictly enforcing one trajectory. Subjects who trained with guidance adopted a strategy that involved faster reaches, required greater control of the system's degrees of freedom, and increased the potential for faster task completion. However, overall improvement of task performance was limited with the new strategy.
Collapse
|