1
|
Wagner L, Jourdan S, Mayer L, Müller C, Bernhard L, Kolb S, Harb F, Jell A, Berlet M, Feussner H, Buxmann P, Knoll A, Wilhelm D. Robotic scrub nurse to anticipate surgical instruments based on real-time laparoscopic video analysis. COMMUNICATIONS MEDICINE 2024; 4:156. [PMID: 39095639 PMCID: PMC11297199 DOI: 10.1038/s43856-024-00581-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/02/2024] [Accepted: 07/25/2024] [Indexed: 08/04/2024] Open
Abstract
BACKGROUND Machine learning and robotics technologies are increasingly being used in the healthcare domain to improve the quality and efficiency of surgeries and to address challenges such as staff shortages. Robotic scrub nurses in particular offer great potential to address staff shortages by assuming nursing tasks such as the handover of surgical instruments. METHODS We introduce a robotic scrub nurse system designed to enhance the quality of surgeries and efficiency of surgical workflows by predicting and delivering the required surgical instruments based on real-time laparoscopic video analysis. We propose a three-stage deep learning architecture consisting of a single frame-, temporal multi frame-, and informed model to anticipate surgical instruments. The anticipation model was trained on a total of 62 laparoscopic cholecystectomies. RESULTS Here, we show that our prediction system can accurately anticipate 71.54% of the surgical instruments required during laparoscopic cholecystectomies in advance, facilitating a smoother surgical workflow and reducing the need for verbal communication. As the instruments in the left working trocar are changed less frequently and according to a standardized procedure, the prediction system works particularly well for this trocar. CONCLUSIONS The robotic scrub nurse thus acts as a mind reader and helps to mitigate staff shortages by taking over a great share of the workload during surgeries while additionally enabling an enhanced process standardization.
Collapse
Affiliation(s)
- Lars Wagner
- Technical University of Munich, TUM School of Medicine and Health, Klinikum rechts der Isar, Research Group MITI, Munich, Germany.
| | - Sara Jourdan
- Technical University of Darmstadt, Software & Digital Business Group, Darmstadt, Germany
| | - Leon Mayer
- Technical University of Munich, TUM School of Medicine and Health, Klinikum rechts der Isar, Research Group MITI, Munich, Germany
| | - Carolin Müller
- Technical University of Munich, TUM School of Medicine and Health, Klinikum rechts der Isar, Research Group MITI, Munich, Germany
| | - Lukas Bernhard
- Technical University of Munich, TUM School of Medicine and Health, Klinikum rechts der Isar, Research Group MITI, Munich, Germany
| | - Sven Kolb
- Technical University of Munich, TUM School of Medicine and Health, Klinikum rechts der Isar, Research Group MITI, Munich, Germany
| | - Farid Harb
- Technical University of Munich, TUM School of Medicine and Health, Klinikum rechts der Isar, Research Group MITI, Munich, Germany
| | - Alissa Jell
- Technical University of Munich, TUM School of Medicine and Health, Klinikum rechts der Isar, Research Group MITI, Munich, Germany
- Technical University of Munich, TUM School of Medicine and Health, Klinikum rechts der Isar, Department of Surgery, Munich, Germany
| | - Maximilian Berlet
- Technical University of Munich, TUM School of Medicine and Health, Klinikum rechts der Isar, Research Group MITI, Munich, Germany
- Technical University of Munich, TUM School of Medicine and Health, Klinikum rechts der Isar, Department of Surgery, Munich, Germany
| | - Hubertus Feussner
- Technical University of Munich, TUM School of Medicine and Health, Klinikum rechts der Isar, Research Group MITI, Munich, Germany
- Technical University of Munich, TUM School of Medicine and Health, Klinikum rechts der Isar, Department of Surgery, Munich, Germany
| | - Peter Buxmann
- Technical University of Darmstadt, Software & Digital Business Group, Darmstadt, Germany
| | - Alois Knoll
- Technical University of Munich, TUM School of Computation, Information and Technology, Chair of Robotics, Artificial Intelligence and Real-Time Systems, Garching, Germany
| | - Dirk Wilhelm
- Technical University of Munich, TUM School of Medicine and Health, Klinikum rechts der Isar, Research Group MITI, Munich, Germany
- Technical University of Munich, TUM School of Medicine and Health, Klinikum rechts der Isar, Department of Surgery, Munich, Germany
- Technical University of Munich, Munich Institute of Robotics and Machine Intelligence, Munich, Germany
| |
Collapse
|
2
|
Wagner L, Kolb S, Looschen C, Bernhard L, Fuchtmann J, Berlet M, Fottner J, Knoll A, Wilhelm D. Versatile end effector for laparoscopic robotic scrub nurse. Int J Comput Assist Radiol Surg 2023; 18:1589-1600. [PMID: 37154830 PMCID: PMC10491531 DOI: 10.1007/s11548-023-02892-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/10/2023] [Accepted: 03/28/2023] [Indexed: 05/10/2023]
Abstract
PURPOSE Integrating robotic scrub nurses in the operating room has the potential to help overcome staff shortages and limited use of available operating capacities in hospitals. Existing approaches of robotic scrub nurses are mainly focused on open surgical procedures, neglecting laparoscopic procedures. Laparoscopic interventions offer great potential for the context-sensitive integration of robotic systems due to possible standardization. However, the first step is to ensure the safe manipulation of laparoscopic instruments. METHODS A robotic platform with a universal gripper system was designed to pick up and place laparoscopic as well as da Vinci[Formula: see text] instruments in an efficient workflow. The robustness of the gripper system was studied using a test protocol, which included a force absorption test to determine the operational safety limits of the design and a grip test to determine the system performance. RESULTS The test protocol shows results regarding force and torque absorption capabilities of the end effector, which are essential when transferring an instrument to the surgeon to enable a robust handover. The grip tests show that the laparoscopic instruments can be safely picked up, manipulated and returned independent of unexpected positional deviations. The gripper system also enables the manipulation of da Vinci[Formula: see text] instruments, opening the door for robot-robot interaction. CONCLUSION Our evaluation tests have shown that our robotic scrub nurse with the universal gripper system can safely and robustly manipulate laparoscopic and da Vinci[Formula: see text] instruments. The system design will continue with the integration of context-sensitive capabilities.
Collapse
Affiliation(s)
- Lars Wagner
- Research Group MITI, University Hospital rechts der Isar, Technical University of Munich, Munich, Germany.
| | - Sven Kolb
- Research Group MITI, University Hospital rechts der Isar, Technical University of Munich, Munich, Germany
| | - Christian Looschen
- Chair of Materials Handling, Material Flow, Logistics, Technical University of Munich, Munich, Germany
| | - Lukas Bernhard
- Research Group MITI, University Hospital rechts der Isar, Technical University of Munich, Munich, Germany
| | - Jonas Fuchtmann
- Research Group MITI, University Hospital rechts der Isar, Technical University of Munich, Munich, Germany
| | - Maximilian Berlet
- Research Group MITI, University Hospital rechts der Isar, Technical University of Munich, Munich, Germany
- Department of Surgery, University Hospital rechts der Isar, Technical University of Munich, Munich, Germany
| | - Johannes Fottner
- Chair of Materials Handling, Material Flow, Logistics, Technical University of Munich, Munich, Germany
| | - Alois Knoll
- Chair of Robotics, Artificial Intelligence and Real-Time Systems, Technical University of Munich, Munich, Germany
| | - Dirk Wilhelm
- Research Group MITI, University Hospital rechts der Isar, Technical University of Munich, Munich, Germany
- Department of Surgery, University Hospital rechts der Isar, Technical University of Munich, Munich, Germany
| |
Collapse
|
3
|
Zhou D, Aoyama Y, Takeyama H, Tadano K, Haraguchi D. Automation of Intraoperative Tool Changing for Robot-Assisted Laparoscopic Surgery. JOURNAL OF ROBOTICS AND MECHATRONICS 2022. [DOI: 10.20965/jrm.2022.p1216] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/23/2022]
Abstract
Intraoperative tool change is a time-consuming and labor-intensive task for robot-assisted laparoscopic surgery. Serial multi-DOF manipulators are potential devices for realizing automatic intraoperative tool changes because of the layout flexibility and motion range, and multi-DOF makes it feasible for the manipulator to access and fetch the surgical tools by itself. However, the direction of the trocar may change because of the soft abdomen, and the lack of a fixed RCM makes it difficult for manipulators to reinsert a new surgical tool through the trocar. This study proposes a system prototype using a 7-DOF manipulator to automatically conduct the intraoperative tool-changing task. The newly designed surgical tool docking station facilitates surgical tool coupling/decoupling by rotating the manipulator’s end effector once. The proposed trocar recognition method with position error compensation is reliable for aligning a new surgical tool to the trocar port, even when the direction of the trocar is changed. The experimental results confirms that the manipulator can accomplish an intraoperative tool changing task without additional assistance or correction from the human.
Collapse
|
4
|
Kim KY, Lee JJ. Design and evaluation of a slave manipulator with roll-pitch-roll wrist and automatic tool loading mechanism in telerobotic surgery. Int J Med Robot 2012; 8:421-35. [PMID: 23081717 DOI: 10.1002/rcs.1458] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 08/09/2012] [Indexed: 11/09/2022]
Abstract
BACKGROUND As there is a shortage of scrub nurses in many hospitals, automatic surgical tool exchanging mechanism without human labour has been studied. Minimally invasive robotic surgeries (MIRS) also require scrub nurses. A surgical tool loading mechanism without a scrub nurse's assistance for MIRS is proposed. Many researchers have developed minimally invasive surgical instruments with a wrist joint that can be movable inside the abdomen. However, implementation of a distal rolling joint on a gripper is rare. METHODS To implement surgical tool exchanging without a scrub nurse's assistance, a slave manipulator and a tool loader were developed to load and unload a surgical tool unit. A surgical tool unit with a roll-pitch-roll wrist was developed. Several experiments were performed to validate the effectiveness of the slave manipulator and the surgical tool unit. RESULTS The slave manipulator and the tool loader were able to successfully unload and load the surgical tool unit without human assistance. The total duration of unloading and loading the surgical tool unit was 97 s. Motion tracking experiments of the distal rolling joint were performed. The maximum positioning error of the step input response was 2°. CONCLUSIONS The advantage of the proposed slave manipulator and tool loader is that other robotic systems or human labour are not needed for surgical tool loading. The feasibility of the distal rolling joint in MIS is verified.
Collapse
Affiliation(s)
- Ki-Young Kim
- Department of Mechanical Engineering, KAIST (Korea Advanced Institute of Science and Technology), South Korea
| | | |
Collapse
|