1
|
Levin I, Rapoport Ferman J, Bar O, Ben Ayoun D, Cohen A, Wolf T. Introducing surgical intelligence in gynecology: Automated identification of key steps in hysterectomy. Int J Gynaecol Obstet 2024; 166:1273-1278. [PMID: 38546527 DOI: 10.1002/ijgo.15490] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/07/2024] [Revised: 02/27/2024] [Accepted: 03/10/2024] [Indexed: 08/16/2024]
Abstract
OBJECTIVE The analysis of surgical videos using artificial intelligence holds great promise for the future of surgery by facilitating the development of surgical best practices, identifying key pitfalls, enhancing situational awareness, and disseminating that information via real-time, intraoperative decision-making. The objective of the present study was to examine the feasibility and accuracy of a novel computer vision algorithm for hysterectomy surgical step identification. METHODS This was a retrospective study conducted on surgical videos of laparoscopic hysterectomies performed in 277 patients in five medical centers. We used a surgical intelligence platform (Theator Inc.) that employs advanced computer vision and AI technology to automatically capture video data during surgery, deidentify, and upload procedures to a secure cloud infrastructure. Videos were manually annotated with sequential steps of surgery by a team of annotation specialists. Subsequently, a computer vision system was trained to perform automated step detection in hysterectomy. Analyzing automated video annotations in comparison to manual human annotations was used to determine accuracy. RESULTS The mean duration of the videos was 103 ± 43 min. Accuracy between AI-based predictions and manual human annotations was 93.1% on average. Accuracy was highest for the dissection and mobilization step (96.9%) and lowest for the adhesiolysis step (70.3%). CONCLUSION The results of the present study demonstrate that a novel AI-based model achieves high accuracy for automated steps identification in hysterectomy. This lays the foundations for the next phase of AI, focused on real-time clinical decision support and prediction of outcome measures, to optimize surgeon workflow and elevate patient care.
Collapse
Affiliation(s)
- Ishai Levin
- Department of Gynecology, Lis Maternity Hospital, Tel Aviv Sourasky Medical Center, Tel Aviv, Israel
- Faculty of Medicine, Tel Aviv University, Tel Aviv, Israel
| | | | - Omri Bar
- Theator Inc, Palo Alto, California, USA
| | | | - Aviad Cohen
- Department of Gynecology, Lis Maternity Hospital, Tel Aviv Sourasky Medical Center, Tel Aviv, Israel
- Faculty of Medicine, Tel Aviv University, Tel Aviv, Israel
| | | |
Collapse
|
2
|
Laterza V, Marchegiani F, Aisoni F, Ammendola M, Schena CA, Lavazza L, Ravaioli C, Carra MC, Costa V, De Franceschi A, De Simone B, de’Angelis N. Smart Operating Room in Digestive Surgery: A Narrative Review. Healthcare (Basel) 2024; 12:1530. [PMID: 39120233 PMCID: PMC11311806 DOI: 10.3390/healthcare12151530] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/30/2024] [Revised: 07/24/2024] [Accepted: 07/29/2024] [Indexed: 08/10/2024] Open
Abstract
The introduction of new technologies in current digestive surgical practice is progressively reshaping the operating room, defining the fourth surgical revolution. The implementation of black boxes and control towers aims at streamlining workflow and reducing surgical error by early identification and analysis, while augmented reality and artificial intelligence augment surgeons' perceptual and technical skills by superimposing three-dimensional models to real-time surgical images. Moreover, the operating room architecture is transitioning toward an integrated digital environment to improve efficiency and, ultimately, patients' outcomes. This narrative review describes the most recent evidence regarding the role of these technologies in transforming the current digestive surgical practice, underlining their potential benefits and drawbacks in terms of efficiency and patients' outcomes, as an attempt to foresee the digestive surgical practice of tomorrow.
Collapse
Affiliation(s)
- Vito Laterza
- Department of Digestive Surgical Oncology and Liver Transplantation, University Hospital of Besançon, 3 Boulevard Alexandre Fleming, 25000 Besancon, France;
| | - Francesco Marchegiani
- Unit of Colorectal and Digestive Surgery, DIGEST Department, Beaujon University Hospital, AP-HP, University of Paris Cité, Clichy, 92110 Paris, France
| | - Filippo Aisoni
- Unit of Emergency Surgery, Department of Surgery, Ferrara University Hospital, 44124 Ferrara, Italy;
| | - Michele Ammendola
- Digestive Surgery Unit, Health of Science Department, University Hospital “R.Dulbecco”, 88100 Catanzaro, Italy;
| | - Carlo Alberto Schena
- Unit of Robotic and Minimally Invasive Surgery, Department of Surgery, Ferrara University Hospital, 44124 Ferrara, Italy; (C.A.S.); (N.d.)
| | - Luca Lavazza
- Hospital Network Coordinator of Azienda Ospedaliero, Universitaria and Azienda USL di Ferrara, 44121 Ferrara, Italy;
| | - Cinzia Ravaioli
- Azienda Ospedaliero, Universitaria di Ferrara, 44121 Ferrara, Italy;
| | - Maria Clotilde Carra
- Rothschild Hospital (AP-HP), 75012 Paris, France;
- INSERM-Sorbonne Paris Cité, Epidemiology and Statistics Research Centre, 75004 Paris, France
| | - Vittore Costa
- Unit of Orthopedics, Humanitas Hospital, 24125 Bergamo, Italy;
| | | | - Belinda De Simone
- Department of Emergency Surgery, Academic Hospital of Villeneuve St Georges, 91560 Villeneuve St. Georges, France;
| | - Nicola de’Angelis
- Unit of Robotic and Minimally Invasive Surgery, Department of Surgery, Ferrara University Hospital, 44124 Ferrara, Italy; (C.A.S.); (N.d.)
- Department of Translational Medicine, University of Ferrara, 44121 Ferrara, Italy
| |
Collapse
|
3
|
Lin W, Hu Y, Fu H, Yang M, Chng CB, Kawasaki R, Chui C, Liu J. Instrument-Tissue Interaction Detection Framework for Surgical Video Understanding. IEEE TRANSACTIONS ON MEDICAL IMAGING 2024; 43:2803-2813. [PMID: 38530715 DOI: 10.1109/tmi.2024.3381209] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 03/28/2024]
Abstract
Instrument-tissue interaction detection task, which helps understand surgical activities, is vital for constructing computer-assisted surgery systems but with many challenges. Firstly, most models represent instrument-tissue interaction in a coarse-grained way which only focuses on classification and lacks the ability to automatically detect instruments and tissues. Secondly, existing works do not fully consider relations between intra- and inter-frame of instruments and tissues. In the paper, we propose to represent instrument-tissue interaction as 〈 instrument class, instrument bounding box, tissue class, tissue bounding box, action class 〉 quintuple and present an Instrument-Tissue Interaction Detection Network (ITIDNet) to detect the quintuple for surgery videos understanding. Specifically, we propose a Snippet Consecutive Feature (SCF) Layer to enhance features by modeling relationships of proposals in the current frame using global context information in the video snippet. We also propose a Spatial Corresponding Attention (SCA) Layer to incorporate features of proposals between adjacent frames through spatial encoding. To reason relationships between instruments and tissues, a Temporal Graph (TG) Layer is proposed with intra-frame connections to exploit relationships between instruments and tissues in the same frame and inter-frame connections to model the temporal information for the same instance. For evaluation, we build a cataract surgery video (PhacoQ) dataset and a cholecystectomy surgery video (CholecQ) dataset. Experimental results demonstrate the promising performance of our model, which outperforms other state-of-the-art models on both datasets.
Collapse
|
4
|
Younis R, Yamlahi A, Bodenstedt S, Scheikl PM, Kisilenko A, Daum M, Schulze A, Wise PA, Nickel F, Mathis-Ullrich F, Maier-Hein L, Müller-Stich BP, Speidel S, Distler M, Weitz J, Wagner M. A surgical activity model of laparoscopic cholecystectomy for co-operation with collaborative robots. Surg Endosc 2024; 38:4316-4328. [PMID: 38872018 PMCID: PMC11289174 DOI: 10.1007/s00464-024-10958-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/15/2024] [Accepted: 05/24/2024] [Indexed: 06/15/2024]
Abstract
BACKGROUND Laparoscopic cholecystectomy is a very frequent surgical procedure. However, in an ageing society, less surgical staff will need to perform surgery on patients. Collaborative surgical robots (cobots) could address surgical staff shortages and workload. To achieve context-awareness for surgeon-robot collaboration, the intraoperative action workflow recognition is a key challenge. METHODS A surgical process model was developed for intraoperative surgical activities including actor, instrument, action and target in laparoscopic cholecystectomy (excluding camera guidance). These activities, as well as instrument presence and surgical phases were annotated in videos of laparoscopic cholecystectomy performed on human patients (n = 10) and on explanted porcine livers (n = 10). The machine learning algorithm Distilled-Swin was trained on our own annotated dataset and the CholecT45 dataset. The validation of the model was conducted using a fivefold cross-validation approach. RESULTS In total, 22,351 activities were annotated with a cumulative duration of 24.9 h of video segments. The machine learning algorithm trained and validated on our own dataset scored a mean average precision (mAP) of 25.7% and a top K = 5 accuracy of 85.3%. With training and validation on our dataset and CholecT45, the algorithm scored a mAP of 37.9%. CONCLUSIONS An activity model was developed and applied for the fine-granular annotation of laparoscopic cholecystectomies in two surgical settings. A machine recognition algorithm trained on our own annotated dataset and CholecT45 achieved a higher performance than training only on CholecT45 and can recognize frequently occurring activities well, but not infrequent activities. The analysis of an annotated dataset allowed for the quantification of the potential of collaborative surgical robots to address the workload of surgical staff. If collaborative surgical robots could grasp and hold tissue, up to 83.5% of the assistant's tissue interacting tasks (i.e. excluding camera guidance) could be performed by robots.
Collapse
Affiliation(s)
- R Younis
- Department for General, Visceral and Transplant Surgery, Heidelberg University Hospital, Heidelberg, Germany
- National Center for Tumor Diseases (NCT), Heidelberg, Germany
- Centre for the Tactile Internet with Human-in-the-Loop (CeTI), TUD Dresden University of Technology, Dresden, Germany
| | - A Yamlahi
- Division of Intelligent Medical Systems (IMSY), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - S Bodenstedt
- Department for Translational Surgical Oncology, National Center for Tumor Diseases, Partner Site Dresden, Dresden, Germany
- Centre for the Tactile Internet with Human-in-the-Loop (CeTI), TUD Dresden University of Technology, Dresden, Germany
| | - P M Scheikl
- Surgical Planning and Robotic Cognition (SPARC), Department Artificial Intelligence in Biomedical Engineering (AIBE), Friedrich-Alexander-University Erlangen-Nürnberg, Erlangen, Germany
| | - A Kisilenko
- Department for General, Visceral and Transplant Surgery, Heidelberg University Hospital, Heidelberg, Germany
- National Center for Tumor Diseases (NCT), Heidelberg, Germany
| | - M Daum
- Centre for the Tactile Internet with Human-in-the-Loop (CeTI), TUD Dresden University of Technology, Dresden, Germany
- Department of Visceral, Thoracic and Vascular Surgery, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD Dresden University of Technology, Fetscherstraße 74, 01307, Dresden, Germany
| | - A Schulze
- Centre for the Tactile Internet with Human-in-the-Loop (CeTI), TUD Dresden University of Technology, Dresden, Germany
- Department of Visceral, Thoracic and Vascular Surgery, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD Dresden University of Technology, Fetscherstraße 74, 01307, Dresden, Germany
| | - P A Wise
- Department for General, Visceral and Transplant Surgery, Heidelberg University Hospital, Heidelberg, Germany
| | - F Nickel
- Department for General, Visceral and Transplant Surgery, Heidelberg University Hospital, Heidelberg, Germany
- Department of General, Visceral and Thoracic Surgery, University Medical Center Hamburg- Eppendorf, Hamburg, Germany
| | - F Mathis-Ullrich
- Surgical Planning and Robotic Cognition (SPARC), Department Artificial Intelligence in Biomedical Engineering (AIBE), Friedrich-Alexander-University Erlangen-Nürnberg, Erlangen, Germany
| | - L Maier-Hein
- National Center for Tumor Diseases (NCT), Heidelberg, Germany
- Division of Intelligent Medical Systems (IMSY), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - B P Müller-Stich
- Department for Abdominal Surgery, University Center for Gastrointestinal and Liver Diseases, Basel, Switzerland
| | - S Speidel
- Department for Translational Surgical Oncology, National Center for Tumor Diseases, Partner Site Dresden, Dresden, Germany
- Centre for the Tactile Internet with Human-in-the-Loop (CeTI), TUD Dresden University of Technology, Dresden, Germany
| | - M Distler
- Department of Visceral, Thoracic and Vascular Surgery, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD Dresden University of Technology, Fetscherstraße 74, 01307, Dresden, Germany
| | - J Weitz
- Centre for the Tactile Internet with Human-in-the-Loop (CeTI), TUD Dresden University of Technology, Dresden, Germany
- Department of Visceral, Thoracic and Vascular Surgery, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD Dresden University of Technology, Fetscherstraße 74, 01307, Dresden, Germany
| | - M Wagner
- Department for General, Visceral and Transplant Surgery, Heidelberg University Hospital, Heidelberg, Germany.
- National Center for Tumor Diseases (NCT), Heidelberg, Germany.
- Department for Translational Surgical Oncology, National Center for Tumor Diseases, Partner Site Dresden, Dresden, Germany.
- Centre for the Tactile Internet with Human-in-the-Loop (CeTI), TUD Dresden University of Technology, Dresden, Germany.
- Department of Visceral, Thoracic and Vascular Surgery, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD Dresden University of Technology, Fetscherstraße 74, 01307, Dresden, Germany.
| |
Collapse
|
5
|
Rau A, Bano S, Jin Y, Azagra P, Morlana J, Kader R, Sanderson E, Matuszewski BJ, Lee JY, Lee DJ, Posner E, Frank N, Elangovan V, Raviteja S, Li Z, Liu J, Lalithkumar S, Islam M, Ren H, Lovat LB, Montiel JMM, Stoyanov D. SimCol3D - 3D reconstruction during colonoscopy challenge. Med Image Anal 2024; 96:103195. [PMID: 38815359 DOI: 10.1016/j.media.2024.103195] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/04/2023] [Revised: 02/08/2024] [Accepted: 05/02/2024] [Indexed: 06/01/2024]
Abstract
Colorectal cancer is one of the most common cancers in the world. While colonoscopy is an effective screening technique, navigating an endoscope through the colon to detect polyps is challenging. A 3D map of the observed surfaces could enhance the identification of unscreened colon tissue and serve as a training platform. However, reconstructing the colon from video footage remains difficult. Learning-based approaches hold promise as robust alternatives, but necessitate extensive datasets. Establishing a benchmark dataset, the 2022 EndoVis sub-challenge SimCol3D aimed to facilitate data-driven depth and pose prediction during colonoscopy. The challenge was hosted as part of MICCAI 2022 in Singapore. Six teams from around the world and representatives from academia and industry participated in the three sub-challenges: synthetic depth prediction, synthetic pose prediction, and real pose prediction. This paper describes the challenge, the submitted methods, and their results. We show that depth prediction from synthetic colonoscopy images is robustly solvable, while pose estimation remains an open research question.
Collapse
Affiliation(s)
- Anita Rau
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS) and Department of Computer Science, University College London, London, UK; Stanford University, Stanford, CA, USA.
| | - Sophia Bano
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS) and Department of Computer Science, University College London, London, UK.
| | - Yueming Jin
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS) and Department of Computer Science, University College London, London, UK; National University of Singapore, Singapore.
| | | | | | - Rawen Kader
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS) and Department of Computer Science, University College London, London, UK
| | - Edward Sanderson
- Computer Vision and Machine Learning (CVML) Group, University of Central Lancashire, Preston, UK
| | - Bogdan J Matuszewski
- Computer Vision and Machine Learning (CVML) Group, University of Central Lancashire, Preston, UK
| | - Jae Young Lee
- Korea Advanced Institute of Science and Technology, Daejeon, Republic of Korea
| | - Dong-Jae Lee
- Korea Advanced Institute of Science and Technology, Daejeon, Republic of Korea
| | | | | | | | - Sista Raviteja
- Indian Institute of Technology Kharagpur, Kharagpur, India
| | - Zhengwen Li
- Key Laboratory for Biomedical Engineering of Ministry of Education, College of Biomedical Engineering & Instrument Science, Zhejiang University, China
| | - Jiquan Liu
- Key Laboratory for Biomedical Engineering of Ministry of Education, College of Biomedical Engineering & Instrument Science, Zhejiang University, China
| | - Seenivasan Lalithkumar
- National University of Singapore, Singapore; The Chinese University of Hong Kong, Hong Kong, China
| | | | - Hongliang Ren
- National University of Singapore, Singapore; The Chinese University of Hong Kong, Hong Kong, China
| | - Laurence B Lovat
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS) and Department of Computer Science, University College London, London, UK
| | | | - Danail Stoyanov
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS) and Department of Computer Science, University College London, London, UK
| |
Collapse
|
6
|
Hitchcock CL, Chapman GJ, Mojzisik CM, Mueller JK, Martin EW. A Concept for Preoperative and Intraoperative Molecular Imaging and Detection for Assessing Extent of Disease of Solid Tumors. Oncol Rev 2024; 18:1409410. [PMID: 39119243 PMCID: PMC11306801 DOI: 10.3389/or.2024.1409410] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/30/2024] [Accepted: 05/28/2024] [Indexed: 08/10/2024] Open
Abstract
The authors propose a concept of "systems engineering," the approach to assessing the extent of diseased tissue (EODT) in solid tumors. We modeled the proof of this concept based on our clinical experience with colorectal carcinoma (CRC) and gastrinoma that included short and long-term survival data of CRC patients. This concept, applicable to various solid tumors, combines resources from surgery, nuclear medicine, radiology, pathology, and oncology needed for preoperative and intraoperative assessments of a patient's EODT. The concept begins with a patient presenting with biopsy-proven cancer. An appropriate preferential locator (PL) is a molecule that preferentially binds to a cancer-related molecular target (i.e., tumor marker) lacking in non-malignant tissue and is the essential element. Detecting the PL after an intravenous injection requires the PL labeling with an appropriate tracer radionuclide, a fluoroprobe, or both. Preoperative imaging of the tracer's signal requires molecular imaging modalities alone or in combination with computerized tomography (CT). These include positron emission tomography (PET), PET/CT, single-photon emission computed tomography (SPECT), SPECT/CT for preoperative imaging, gamma cameras for intraoperative imaging, and gamma-detecting probes for precise localization. Similarly, fluorescent-labeled PLs require appropriate cameras and probes. This approach provides the surgeon with real-time information needed for R0 resection.
Collapse
Affiliation(s)
- Charles L. Hitchcock
- Department of Pathology, College of Medicine, The Ohio State University, Columbus, OH, United States
- Actis Medical, LLC, Powell, OH, United States
| | - Gregg J. Chapman
- Actis Medical, LLC, Powell, OH, United States
- Department of Electrical and Computer Engineering, College of Engineering, The Ohio State University, Columbus, OH, United States
| | | | | | - Edward W. Martin
- Actis Medical, LLC, Powell, OH, United States
- Division of Surgical Oncology, Department of Surgery, College of Medicine, The Ohio State University, Columbus, OH, United States
| |
Collapse
|
7
|
Jurosch F, Wagner L, Jell A, Islertas E, Wilhelm D, Berlet M. Extra-abdominal trocar and instrument detection for enhanced surgical workflow understanding. Int J Comput Assist Radiol Surg 2024:10.1007/s11548-024-03220-0. [PMID: 39008232 DOI: 10.1007/s11548-024-03220-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/09/2024] [Accepted: 06/24/2024] [Indexed: 07/16/2024]
Abstract
PURPOSE Video-based intra-abdominal instrument tracking for laparoscopic surgeries is a common research area. However, the tracking can only be done with instruments that are actually visible in the laparoscopic image. By using extra-abdominal cameras to detect trocars and classify their occupancy state, additional information about the instrument location, whether an instrument is still in the abdomen or not, can be obtained. This can enhance laparoscopic workflow understanding and enrich already existing intra-abdominal solutions. METHODS A data set of four laparoscopic surgeries recorded with two time-synchronized extra-abdominal 2D cameras was generated. The preprocessed and annotated data were used to train a deep learning-based network architecture consisting of a trocar detection, a centroid tracker and a temporal model to provide the occupancy state of all trocars during the surgery. RESULTS The trocar detection model achieves an F1 score of 95.06 ± 0.88 % . The prediction of the occupancy state yields an F1 score of 89.29 ± 5.29 % , providing a first step towards enhanced surgical workflow understanding. CONCLUSION The current method shows promising results for the extra-abdominal tracking of trocars and their occupancy state. Future advancements include the enlargement of the data set and incorporation of intra-abdominal imaging to facilitate accurate assignment of instruments to trocars.
Collapse
Affiliation(s)
- Franziska Jurosch
- Research Group MITI, Klinikum rechts der Isar, TUM School of Medicine and Health, Technical University of Munich, Munich, Germany.
| | - Lars Wagner
- Research Group MITI, Klinikum rechts der Isar, TUM School of Medicine and Health, Technical University of Munich, Munich, Germany
| | - Alissa Jell
- Research Group MITI, Klinikum rechts der Isar, TUM School of Medicine and Health, Technical University of Munich, Munich, Germany
- Department of Surgery, Klinikum rechts der Isar, TUM School of Medicine and Health, Technical University of Munich, Munich, Germany
| | - Esra Islertas
- Research Group MITI, Klinikum rechts der Isar, TUM School of Medicine and Health, Technical University of Munich, Munich, Germany
| | - Dirk Wilhelm
- Research Group MITI, Klinikum rechts der Isar, TUM School of Medicine and Health, Technical University of Munich, Munich, Germany
- Department of Surgery, Klinikum rechts der Isar, TUM School of Medicine and Health, Technical University of Munich, Munich, Germany
| | - Maximilian Berlet
- Research Group MITI, Klinikum rechts der Isar, TUM School of Medicine and Health, Technical University of Munich, Munich, Germany
- Department of Surgery, Klinikum rechts der Isar, TUM School of Medicine and Health, Technical University of Munich, Munich, Germany
| |
Collapse
|
8
|
Wagner L, Schneider DN, Mayer L, Jell A, Müller C, Lenz A, Knoll A, Wilhelm D. Towards multimodal graph neural networks for surgical instrument anticipation. Int J Comput Assist Radiol Surg 2024:10.1007/s11548-024-03226-8. [PMID: 38985412 DOI: 10.1007/s11548-024-03226-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/10/2024] [Accepted: 06/25/2024] [Indexed: 07/11/2024]
Abstract
PURPOSE Decision support systems and context-aware assistance in the operating room have emerged as the key clinical applications supporting surgeons in their daily work and are generally based on single modalities. The model- and knowledge-based integration of multimodal data as a basis for decision support systems that can dynamically adapt to the surgical workflow has not yet been established. Therefore, we propose a knowledge-enhanced method for fusing multimodal data for anticipation tasks. METHODS We developed a holistic, multimodal graph-based approach combining imaging and non-imaging information in a knowledge graph representing the intraoperative scene of a surgery. Node and edge features of the knowledge graph are extracted from suitable data sources in the operating room using machine learning. A spatiotemporal graph neural network architecture subsequently allows for interpretation of relational and temporal patterns within the knowledge graph. We apply our approach to the downstream task of instrument anticipation while presenting a suitable modeling and evaluation strategy for this task. RESULTS Our approach achieves an F1 score of 66.86% in terms of instrument anticipation, allowing for a seamless surgical workflow and adding a valuable impact for surgical decision support systems. A resting recall of 63.33% indicates the non-prematurity of the anticipations. CONCLUSION This work shows how multimodal data can be combined with the topological properties of an operating room in a graph-based approach. Our multimodal graph architecture serves as a basis for context-sensitive decision support systems in laparoscopic surgery considering a comprehensive intraoperative operating scene.
Collapse
Affiliation(s)
- Lars Wagner
- Technical University of Munich, TUM School of Medicine and Health, Klinikum rechts der Isar, Research Group MITI, Munich, Germany.
| | - Dennis N Schneider
- Technical University of Munich, TUM School of Medicine and Health, Klinikum rechts der Isar, Research Group MITI, Munich, Germany
| | - Leon Mayer
- Technical University of Munich, TUM School of Medicine and Health, Klinikum rechts der Isar, Research Group MITI, Munich, Germany
| | - Alissa Jell
- Technical University of Munich, TUM School of Medicine and Health, Klinikum rechts der Isar, Research Group MITI, Munich, Germany
- Technical University of Munich, TUM School of Medicine and Health, Klinikum rechts der Isar, Department of Surgery, Munich, Germany
| | - Carolin Müller
- Technical University of Munich, TUM School of Medicine and Health, Klinikum rechts der Isar, Research Group MITI, Munich, Germany
| | - Alexander Lenz
- Technical University of Munich, TUM School of Computation, Information and Technology, Chair of Robotics, Artificial Intelligence and Real-Time Systems, Munich, Germany
| | - Alois Knoll
- Technical University of Munich, TUM School of Computation, Information and Technology, Chair of Robotics, Artificial Intelligence and Real-Time Systems, Munich, Germany
| | - Dirk Wilhelm
- Technical University of Munich, TUM School of Medicine and Health, Klinikum rechts der Isar, Research Group MITI, Munich, Germany
- Technical University of Munich, TUM School of Medicine and Health, Klinikum rechts der Isar, Department of Surgery, Munich, Germany
| |
Collapse
|
9
|
You J, Cai H, Wang Y, Bian A, Cheng K, Meng L, Wang X, Gao P, Chen S, Cai Y, Peng B. Artificial intelligence automated surgical phases recognition in intraoperative videos of laparoscopic pancreatoduodenectomy. Surg Endosc 2024:10.1007/s00464-024-10916-6. [PMID: 38958719 DOI: 10.1007/s00464-024-10916-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/04/2024] [Accepted: 05/05/2024] [Indexed: 07/04/2024]
Abstract
BACKGROUND Laparoscopic pancreatoduodenectomy (LPD) is one of the most challenging operations and has a long learning curve. Artificial intelligence (AI) automated surgical phase recognition in intraoperative videos has many potential applications in surgical education, helping shorten the learning curve, but no study has made this breakthrough in LPD. Herein, we aimed to build AI models to recognize the surgical phase in LPD and explore the performance characteristics of AI models. METHODS Among 69 LPD videos from a single surgical team, we used 42 in the building group to establish the models and used the remaining 27 videos in the analysis group to assess the models' performance characteristics. We annotated 13 surgical phases of LPD, including 4 key phases and 9 necessary phases. Two minimal invasive pancreatic surgeons annotated all the videos. We built two AI models for the key phase and necessary phase recognition, based on convolutional neural networks. The overall performance of the AI models was determined mainly by mean average precision (mAP). RESULTS Overall mAPs of the AI models in the test set of the building group were 89.7% and 84.7% for key phases and necessary phases, respectively. In the 27-video analysis group, overall mAPs were 86.8% and 71.2%, with maximum mAPs of 98.1% and 93.9%. We found commonalities between the error of model recognition and the differences of surgeon annotation, and the AI model exhibited bad performance in cases with anatomic variation or lesion involvement with adjacent organs. CONCLUSIONS AI automated surgical phase recognition can be achieved in LPD, with outstanding performance in selective cases. This breakthrough may be the first step toward AI- and video-based surgical education in more complex surgeries.
Collapse
Affiliation(s)
- Jiaying You
- WestChina-California Research Center for Predictive Intervention, Sichuan University West China Hospital, Chengdu, China
- Division of Pancreatic Surgery, Department of General Surgery, Sichuan University West China Hospital, No. 37, Guoxue Alley, Chengdu, 610041, China
| | - He Cai
- Division of Pancreatic Surgery, Department of General Surgery, Sichuan University West China Hospital, No. 37, Guoxue Alley, Chengdu, 610041, China
| | - Yuxian Wang
- Chengdu Withai Innovations Technology Company, Chengdu, China
| | - Ang Bian
- College of Computer Science, Sichuan University, Chengdu, China
| | - Ke Cheng
- Division of Pancreatic Surgery, Department of General Surgery, Sichuan University West China Hospital, No. 37, Guoxue Alley, Chengdu, 610041, China
| | - Lingwei Meng
- Division of Pancreatic Surgery, Department of General Surgery, Sichuan University West China Hospital, No. 37, Guoxue Alley, Chengdu, 610041, China
| | - Xin Wang
- Division of Pancreatic Surgery, Department of General Surgery, Sichuan University West China Hospital, No. 37, Guoxue Alley, Chengdu, 610041, China
| | - Pan Gao
- Division of Pancreatic Surgery, Department of General Surgery, Sichuan University West China Hospital, No. 37, Guoxue Alley, Chengdu, 610041, China
| | - Sirui Chen
- Mianyang Central Hospital, School of Medicine University of Electronic Science and Technology of China, Mianyang, China
| | - Yunqiang Cai
- Division of Pancreatic Surgery, Department of General Surgery, Sichuan University West China Hospital, No. 37, Guoxue Alley, Chengdu, 610041, China.
| | - Bing Peng
- Division of Pancreatic Surgery, Department of General Surgery, Sichuan University West China Hospital, No. 37, Guoxue Alley, Chengdu, 610041, China.
| |
Collapse
|
10
|
Mascagni P, Alapatt D, Sestini L, Yu T, Alfieri S, Morales-Conde S, Padoy N, Perretta S. Applications of artificial intelligence in surgery: clinical, technical, and governance considerations. Cir Esp 2024; 102 Suppl 1:S66-S71. [PMID: 38704146 DOI: 10.1016/j.cireng.2024.04.009] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/25/2024] [Accepted: 04/29/2024] [Indexed: 05/06/2024]
Abstract
Artificial intelligence (AI) will power many of the tools in the armamentarium of digital surgeons. AI methods and surgical proof-of-concept flourish, but we have yet to witness clinical translation and value. Here we exemplify the potential of AI in the care pathway of colorectal cancer patients and discuss clinical, technical, and governance considerations of major importance for the safe translation of surgical AI for the benefit of our patients and practices.
Collapse
Affiliation(s)
- Pietro Mascagni
- IHU Strasbourg, Strasbourg, France; Fondazione Policlinico Universitario A. Gemelli IRCCS, Rome, Italy; Università Cattolica del Sacro Cuore, Rome, Italy.
| | - Deepak Alapatt
- University of Strasbourg, CNRS, INSERM, ICube, UMR7357, Strasbourg, France
| | - Luca Sestini
- University of Strasbourg, CNRS, INSERM, ICube, UMR7357, Strasbourg, France
| | - Tong Yu
- University of Strasbourg, CNRS, INSERM, ICube, UMR7357, Strasbourg, France
| | - Sergio Alfieri
- Fondazione Policlinico Universitario A. Gemelli IRCCS, Rome, Italy; Università Cattolica del Sacro Cuore, Rome, Italy
| | | | - Nicolas Padoy
- IHU Strasbourg, Strasbourg, France; University of Strasbourg, CNRS, INSERM, ICube, UMR7357, Strasbourg, France
| | - Silvana Perretta
- IHU Strasbourg, Strasbourg, France; IRCAD, Research Institute Against Digestive Cancer, Strasbourg, France; Nouvel Hôpital Civil, Hôpitaux Universitaires de Strasbourg, Strasbourg, France
| |
Collapse
|
11
|
Madani A, Liu Y, Pryor A, Altieri M, Hashimoto DA, Feldman L. SAGES surgical data science task force: enhancing surgical innovation, education and quality improvement through data science. Surg Endosc 2024; 38:3489-3493. [PMID: 38831213 DOI: 10.1007/s00464-024-10921-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/13/2024] [Accepted: 05/05/2024] [Indexed: 06/05/2024]
Affiliation(s)
- Amin Madani
- Department of Surgery, University of Toronto, Toronto, ON, Canada.
| | - Yao Liu
- Department of Surgery, Brown University, Providence, RI, USA
| | - Aurora Pryor
- Department of Surgery, Northwell Health, New York, NY, USA
| | - Maria Altieri
- Department of Surgery, Department of Computer and Information Science, University of Pennsylvania, Philadelphia, PA, USA
| | - Daniel A Hashimoto
- Department of Surgery, Department of Computer and Information Science, University of Pennsylvania, Philadelphia, PA, USA
| | - Liane Feldman
- Department of Surgery, McGill University, Montreal, QC, Canada
| |
Collapse
|
12
|
Peng H, Lin S, King D, Su YH, Abuzeid WM, Bly RA, Moe KS, Hannaford B. Reducing annotating load: Active learning with synthetic images in surgical instrument segmentation. Med Image Anal 2024; 97:103246. [PMID: 38943835 DOI: 10.1016/j.media.2024.103246] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/29/2022] [Revised: 05/28/2024] [Accepted: 06/17/2024] [Indexed: 07/01/2024]
Abstract
Accurate instrument segmentation in the endoscopic vision of minimally invasive surgery is challenging due to complex instruments and environments. Deep learning techniques have shown competitive performance in recent years. However, deep learning usually requires a large amount of labeled data to achieve accurate prediction, which poses a significant workload. To alleviate this workload, we propose an active learning-based framework to generate synthetic images for efficient neural network training. In each active learning iteration, a small number of informative unlabeled images are first queried by active learning and manually labeled. Next, synthetic images are generated based on these selected images. The instruments and backgrounds are cropped out and randomly combined with blending and fusion near the boundary. The proposed method leverages the advantage of both active learning and synthetic images. The effectiveness of the proposed method is validated on two sinus surgery datasets and one intraabdominal surgery dataset. The results indicate a considerable performance improvement, especially when the size of the annotated dataset is small. All the code is open-sourced at: https://github.com/HaonanPeng/active_syn_generator.
Collapse
Affiliation(s)
- Haonan Peng
- University of Washington, 185 E Stevens Way NE AE100R, Seattle, WA 98195, USA.
| | - Shan Lin
- University of California San Diego, 9500 Gilman Dr, La Jolla, CA 92093, USA
| | - Daniel King
- University of Washington, 185 E Stevens Way NE AE100R, Seattle, WA 98195, USA
| | - Yun-Hsuan Su
- Mount Holyoke College, 50 College St, South Hadley, MA 01075, USA
| | - Waleed M Abuzeid
- University of Washington, 185 E Stevens Way NE AE100R, Seattle, WA 98195, USA
| | - Randall A Bly
- University of Washington, 185 E Stevens Way NE AE100R, Seattle, WA 98195, USA
| | - Kris S Moe
- University of Washington, 185 E Stevens Way NE AE100R, Seattle, WA 98195, USA
| | - Blake Hannaford
- University of Washington, 185 E Stevens Way NE AE100R, Seattle, WA 98195, USA
| |
Collapse
|
13
|
Berlet M, Jell A, Wagner L, Bernhard L, Fuchtmann J, Wegener L, Feussner H, Friess H, Wilhelm D. Model-based individual life-spanning documentation in visceral surgery: a proof of concept. Int J Comput Assist Radiol Surg 2024:10.1007/s11548-024-03214-y. [PMID: 38884892 DOI: 10.1007/s11548-024-03214-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/08/2024] [Accepted: 06/04/2024] [Indexed: 06/18/2024]
Abstract
INTRODUCTION Surgical documentation has many implications. However, its primary function is to transfer information about surgical procedures to other medical professionals. Thereby, written reports describing procedures in detail are the current standard, impeding comprehensive understanding of patient-individual life-spanning surgical course, especially if surgeries are performed at a timely distance and in diverse facilities. Therefore, we developed a novel model-based approach for documentation of visceral surgeries, denoted as 'Surgical Documentation Markup-Modeling' (SDM-M). MATERIAL AND METHODS For scientific evaluation, we developed a web-based prototype software allowing for creating hierarchical anatomical models that can be modified by individual surgery-related markup information. Thus, a patient's cumulated 'surgical load' can be displayed on a timeline deploying interactive anatomical 3D models. To evaluate the possible impact on daily clinical routine, we performed an evaluation study with 24 surgeons and advanced medical students, elaborating on simulated complex surgical cases, once with classic written reports and once with our prototypical SDM-M software. RESULTS Leveraging SDM-M in an experimental environment reduced the time needed for elaborating simulated complex surgical cases from 354 ± 85 s with the classic approach to 277 ± 128 s. (p = 0.00109) The perceived task load measured by the Raw NASA-TLX was reduced significantly (p = 0.00003) with decreased mental (p = 0.00004) and physical (p = 0.01403) demand. Also, time demand (p = 0.00041), performance (p = 0.00161), effort (p = 0.00024), and frustration (p = 0.00031) were improved significantly. DISCUSSION Model-based approaches for life-spanning surgical documentation could improve the daily clinical elaboration and understanding of complex cases in visceral surgery. Besides reduced workload and time sparing, even a more structured assessment of individual surgical cases could foster improved planning of further surgeries, information transfer, and even scientific evaluation, considering the cumulative 'surgical load.' CONCLUSION Life-spanning model-based documentation of visceral surgical cases could significantly improve surgery and workload.
Collapse
Affiliation(s)
- Maximilian Berlet
- TUM School of Medicine and Health, Klinikum rechts der Isar, Research Group MITI, Klinikum rechts der Isar, Technical University of Munich, Ismaninger Str. 22, 81675, Munich, Germany.
- TUM School of Medicine and Health, Klinikum rechts der Isar, Department of Surgery, Klinikum rechts der Isar, Technical University of Munich, Ismaninger Str. 22, 81675, Munich, Germany.
| | - Alissa Jell
- TUM School of Medicine and Health, Klinikum rechts der Isar, Research Group MITI, Klinikum rechts der Isar, Technical University of Munich, Ismaninger Str. 22, 81675, Munich, Germany
- TUM School of Medicine and Health, Klinikum rechts der Isar, Department of Surgery, Klinikum rechts der Isar, Technical University of Munich, Ismaninger Str. 22, 81675, Munich, Germany
| | - Lars Wagner
- TUM School of Medicine and Health, Klinikum rechts der Isar, Research Group MITI, Klinikum rechts der Isar, Technical University of Munich, Ismaninger Str. 22, 81675, Munich, Germany
| | - Lukas Bernhard
- TUM School of Medicine and Health, Klinikum rechts der Isar, Research Group MITI, Klinikum rechts der Isar, Technical University of Munich, Ismaninger Str. 22, 81675, Munich, Germany
| | - Jonas Fuchtmann
- TUM School of Medicine and Health, Klinikum rechts der Isar, Research Group MITI, Klinikum rechts der Isar, Technical University of Munich, Ismaninger Str. 22, 81675, Munich, Germany
| | - Luca Wegener
- TUM School of Medicine and Health, Klinikum rechts der Isar, Research Group MITI, Klinikum rechts der Isar, Technical University of Munich, Ismaninger Str. 22, 81675, Munich, Germany
| | - Hubertus Feussner
- TUM School of Medicine and Health, Klinikum rechts der Isar, Research Group MITI, Klinikum rechts der Isar, Technical University of Munich, Ismaninger Str. 22, 81675, Munich, Germany
- TUM School of Medicine and Health, Klinikum rechts der Isar, Department of Surgery, Klinikum rechts der Isar, Technical University of Munich, Ismaninger Str. 22, 81675, Munich, Germany
| | - Helmut Friess
- TUM School of Medicine and Health, Klinikum rechts der Isar, Research Group MITI, Klinikum rechts der Isar, Technical University of Munich, Ismaninger Str. 22, 81675, Munich, Germany
- TUM School of Medicine and Health, Klinikum rechts der Isar, Department of Surgery, Klinikum rechts der Isar, Technical University of Munich, Ismaninger Str. 22, 81675, Munich, Germany
| | - Dirk Wilhelm
- TUM School of Medicine and Health, Klinikum rechts der Isar, Research Group MITI, Klinikum rechts der Isar, Technical University of Munich, Ismaninger Str. 22, 81675, Munich, Germany
- TUM School of Medicine and Health, Klinikum rechts der Isar, Department of Surgery, Klinikum rechts der Isar, Technical University of Munich, Ismaninger Str. 22, 81675, Munich, Germany
| |
Collapse
|
14
|
Fuchtmann J, Riedel T, Berlet M, Jell A, Wegener L, Wagner L, Graf S, Wilhelm D, Ostler-Mildner D. Audio-based event detection in the operating room. Int J Comput Assist Radiol Surg 2024:10.1007/s11548-024-03211-1. [PMID: 38862745 DOI: 10.1007/s11548-024-03211-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/09/2024] [Accepted: 06/03/2024] [Indexed: 06/13/2024]
Abstract
PURPOSE Even though workflow analysis in the operating room has come a long way, current systems are still limited to research. In the quest for a robust, universal setup, hardly any attention has been given to the dimension of audio despite its numerous advantages, such as low costs, location, and sight independence, or little required processing power. METHODOLOGY We present an approach for audio-based event detection that solely relies on two microphones capturing the sound in the operating room. Therefore, a new data set was created with over 63 h of audio recorded and annotated at the University Hospital rechts der Isar. Sound files were labeled, preprocessed, augmented, and subsequently converted to log-mel-spectrograms that served as a visual input for an event classification using pretrained convolutional neural networks. RESULTS Comparing multiple architectures, we were able to show that even lightweight models, such as MobileNet, can already provide promising results. Data augmentation additionally improved the classification of 11 defined classes, including inter alia different types of coagulation, operating table movements as well as an idle class. With the newly created audio data set, an overall accuracy of 90%, a precision of 91% and a F1-score of 91% were achieved, demonstrating the feasibility of an audio-based event recognition in the operating room. CONCLUSION With this first proof of concept, we demonstrated that audio events can serve as a meaningful source of information that goes beyond spoken language and can easily be integrated into future workflow recognition pipelines using computational inexpensive architectures.
Collapse
Affiliation(s)
- Jonas Fuchtmann
- Research Group MITI, Klinikum rechts der Isar, TUM School of Medicine and Health, Technical University of Munich, Munich, Germany.
- Department of Surgery, Klinikum rechts der Isar, TUM School of Medicine and Health, Technical University of Munich, Munich, Germany.
| | - Thomas Riedel
- Research Group MITI, Klinikum rechts der Isar, TUM School of Medicine and Health, Technical University of Munich, Munich, Germany
| | - Maximilian Berlet
- Research Group MITI, Klinikum rechts der Isar, TUM School of Medicine and Health, Technical University of Munich, Munich, Germany
- Department of Surgery, Klinikum rechts der Isar, TUM School of Medicine and Health, Technical University of Munich, Munich, Germany
| | - Alissa Jell
- Research Group MITI, Klinikum rechts der Isar, TUM School of Medicine and Health, Technical University of Munich, Munich, Germany
- Department of Surgery, Klinikum rechts der Isar, TUM School of Medicine and Health, Technical University of Munich, Munich, Germany
| | - Luca Wegener
- Research Group MITI, Klinikum rechts der Isar, TUM School of Medicine and Health, Technical University of Munich, Munich, Germany
| | - Lars Wagner
- Research Group MITI, Klinikum rechts der Isar, TUM School of Medicine and Health, Technical University of Munich, Munich, Germany
| | - Simone Graf
- University Hospital of Hearing, Speech and Voice Disorders, Medical University of Innsbruck, Innsbruck, Austria
| | - Dirk Wilhelm
- Research Group MITI, Klinikum rechts der Isar, TUM School of Medicine and Health, Technical University of Munich, Munich, Germany
- Department of Surgery, Klinikum rechts der Isar, TUM School of Medicine and Health, Technical University of Munich, Munich, Germany
| | - Daniel Ostler-Mildner
- Research Group MITI, Klinikum rechts der Isar, TUM School of Medicine and Health, Technical University of Munich, Munich, Germany
| |
Collapse
|
15
|
Batić D, Holm F, Özsoy E, Czempiel T, Navab N. EndoViT: pretraining vision transformers on a large collection of endoscopic images. Int J Comput Assist Radiol Surg 2024; 19:1085-1091. [PMID: 38570373 PMCID: PMC11178556 DOI: 10.1007/s11548-024-03091-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/15/2024] [Accepted: 02/28/2024] [Indexed: 04/05/2024]
Abstract
PURPOSE Automated endoscopy video analysis is essential for assisting surgeons during medical procedures, but it faces challenges due to complex surgical scenes and limited annotated data. Large-scale pretraining has shown great success in natural language processing and computer vision communities in recent years. These approaches reduce the need for annotated data, which is of great interest in the medical domain. In this work, we investigate endoscopy domain-specific self-supervised pretraining on large collections of data. METHODS To this end, we first collect Endo700k, the largest publicly available corpus of endoscopic images, extracted from nine public Minimally Invasive Surgery (MIS) datasets. Endo700k comprises more than 700,000 images. Next, we introduce EndoViT, an endoscopy-pretrained Vision Transformer (ViT), and evaluate it on a diverse set of surgical downstream tasks. RESULTS Our findings indicate that domain-specific pretraining with EndoViT yields notable advantages in complex downstream tasks. In the case of action triplet recognition, our approach outperforms ImageNet pretraining. In semantic segmentation, we surpass the state-of-the-art (SOTA) performance. These results demonstrate the effectiveness of our domain-specific pretraining approach in addressing the challenges of automated endoscopy video analysis. CONCLUSION Our study contributes to the field of medical computer vision by showcasing the benefits of domain-specific large-scale self-supervised pretraining for vision transformers. We release both our code and pretrained models to facilitate further research in this direction: https://github.com/DominikBatic/EndoViT .
Collapse
Affiliation(s)
- Dominik Batić
- Chair for Computer Aided Medical Procedures, Technical University Munich, Munich, Germany
| | - Felix Holm
- Chair for Computer Aided Medical Procedures, Technical University Munich, Munich, Germany.
- Carl Zeiss AG, Munich, Germany.
| | - Ege Özsoy
- Chair for Computer Aided Medical Procedures, Technical University Munich, Munich, Germany
| | - Tobias Czempiel
- Chair for Computer Aided Medical Procedures, Technical University Munich, Munich, Germany
| | - Nassir Navab
- Chair for Computer Aided Medical Procedures, Technical University Munich, Munich, Germany
| |
Collapse
|
16
|
Malukhin K, Rabczuk T, Ehmann K, Verta MJ. Kirchhoff's law-based velocity-controlled motion models to predict real-time cutting forces in minimally invasive surgeries. J Mech Behav Biomed Mater 2024; 154:106523. [PMID: 38554581 DOI: 10.1016/j.jmbbm.2024.106523] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/29/2023] [Revised: 03/06/2024] [Accepted: 03/21/2024] [Indexed: 04/01/2024]
Abstract
A theoretical framework, united by a "system effect" is formulated to model the cutting/haptic force evolution at the cutting edge of a surgical cutting instrument during its penetration into soft biological tissue in minimally invasive surgery. Other cutting process responses, including tissue fracture force, friction force, and damping, are predicted by the model as well. The model is based on a velocity-controlled formulation of the corresponding equations of motion, derived for a surgical cutting instrument and tissue based on Kirchhoff's fundamental energy conservation law. It provides nearly zero residues (absolute errors) in the equations of motion balances. In addition, concurrent closing relationships for the fracture force, friction coefficient, friction force, process damping, strain rate function (a constitutive tissue model), and their implementation within the proposed theoretical framework are established. The advantage of the method is its ability to make precise real-time predictions of the aperiodic fluctuating evolutions of the cutting forces and the other process responses. It allows for the robust modeling of the interactions between a medical instrument and a nonlinear viscoelastic tissue under any physically feasible working conditions. The cutting process model was partially qualitatively verified through numerical simulations and by comparing the computed cutting forces with experimentally measured values during robotic uniaxial biopsy needle constant velocity insertion into artificial gel tissue, obtained from previous experimental research. The comparison has shown a qualitatively similar adequate trend in the evolution of the experimentally measured and numerically predicted cutting forces during insertion of the needle.
Collapse
Affiliation(s)
- Kostyantyn Malukhin
- Northwestern University, Department of Mechanical Engineering, McCormick School of Engineering, 2145 Sheridan Road, Evanston, IL, 60208, USA.
| | - Timon Rabczuk
- Bauhaus University, Department of Computational Mechanics, School of Civil Engineering, Marienstrasse 15, Weimar, 99423, Germany
| | - Kornel Ehmann
- Northwestern University, Department of Mechanical Engineering, McCormick School of Engineering, 2145 Sheridan Road, Evanston, IL, 60208, USA
| | - Michael J Verta
- Northwestern University, Feinberg School of Medicine, Department of Surgery, 420 E. Superior St., Chicago, IL, 60611, USA
| |
Collapse
|
17
|
Yamada Y, Colan J, Davila A, Hasegawa Y. Multimodal semi-supervised learning for online recognition of multi-granularity surgical workflows. Int J Comput Assist Radiol Surg 2024; 19:1075-1083. [PMID: 38558289 PMCID: PMC11178653 DOI: 10.1007/s11548-024-03101-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/29/2024] [Accepted: 03/04/2024] [Indexed: 04/04/2024]
Abstract
Purpose Surgical workflow recognition is a challenging task that requires understanding multiple aspects of surgery, such as gestures, phases, and steps. However, most existing methods focus on single-task or single-modal models and rely on costly annotations for training. To address these limitations, we propose a novel semi-supervised learning approach that leverages multimodal data and self-supervision to create meaningful representations for various surgical tasks. Methods Our representation learning approach conducts two processes. In the first stage, time contrastive learning is used to learn spatiotemporal visual features from video data, without any labels. In the second stage, multimodal VAE fuses the visual features with kinematic data to obtain a shared representation, which is fed into recurrent neural networks for online recognition. Results Our method is evaluated on two datasets: JIGSAWS and MISAW. We confirmed that it achieved comparable or better performance in multi-granularity workflow recognition compared to fully supervised models specialized for each task. On the JIGSAWS Suturing dataset, we achieve a gesture recognition accuracy of 83.3%. In addition, our model is more efficient in annotation usage, as it can maintain high performance with only half of the labels. On the MISAW dataset, we achieve 84.0% AD-Accuracy in phase recognition and 56.8% AD-Accuracy in step recognition. Conclusion Our multimodal representation exhibits versatility across various surgical tasks and enhances annotation efficiency. This work has significant implications for real-time decision-making systems within the operating room.
Collapse
Affiliation(s)
- Yutaro Yamada
- Department of Micro-Nano Mechanical Science and Engineering, Nagoya University, Furo-cho, Chikusa-ku, Nagoya, Aichi, 464-8603, Japan.
| | - Jacinto Colan
- Department of Micro-Nano Mechanical Science and Engineering, Nagoya University, Furo-cho, Chikusa-ku, Nagoya, Aichi, 464-8603, Japan
| | - Ana Davila
- Institutes of Innovation for Future Society, Nagoya University, Furo-cho, Chikusa-ku, Nagoya, Aichi, 464-8601, Japan
| | - Yasuhisa Hasegawa
- Department of Micro-Nano Mechanical Science and Engineering, Nagoya University, Furo-cho, Chikusa-ku, Nagoya, Aichi, 464-8603, Japan
| |
Collapse
|
18
|
Venkatesh DK, Rivoir D, Pfeiffer M, Kolbinger F, Distler M, Weitz J, Speidel S. Exploring semantic consistency in unpaired image translation to generate data for surgical applications. Int J Comput Assist Radiol Surg 2024; 19:985-993. [PMID: 38407730 DOI: 10.1007/s11548-024-03079-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/22/2024] [Accepted: 02/14/2024] [Indexed: 02/27/2024]
Abstract
PURPOSE In surgical computer vision applications, data privacy and expert annotation challenges impede the acquisition of labeled training data. Unpaired image-to-image translation techniques have been explored to automatically generate annotated datasets by translating synthetic images into a realistic domain. The preservation of structure and semantic consistency, i.e., per-class distribution during translation, poses a significant challenge, particularly in cases of semantic distributional mismatch. METHOD This study empirically investigates various translation methods for generating data in surgical applications, explicitly focusing on semantic consistency. Through our analysis, we introduce a novel and simple combination of effective approaches, which we call ConStructS. The defined losses within this approach operate on multiple image patches and spatial resolutions during translation. RESULTS Various state-of-the-art models were extensively evaluated on two challenging surgical datasets. With two different evaluation schemes, the semantic consistency and the usefulness of the translated images on downstream semantic segmentation tasks were evaluated. The results demonstrate the effectiveness of the ConStructS method in minimizing semantic distortion, with images generated by this model showing superior utility for downstream training. CONCLUSION In this study, we tackle semantic inconsistency in unpaired image translation for surgical applications with minimal labeled data. The simple model (ConStructS) enhances consistency during translation and serves as a practical way of generating fully labeled and semantically consistent datasets at minimal cost. Our code is available at https://gitlab.com/nct_tso_public/constructs .
Collapse
Affiliation(s)
- Danush Kumar Venkatesh
- Department of Translational Surgical Oncology, National Centre for Tumor Diseases(NCT/UCC), Dresden, 01307, Germany.
- SECAI, TU Dresden, Dresden, Germany.
- Department of Visceral, Thoracic and Vascular Surgery, University Hospital and Faculty of Medicine, TU Dresden, 01307, Dresden, Germany.
| | - Dominik Rivoir
- Department of Translational Surgical Oncology, National Centre for Tumor Diseases(NCT/UCC), Dresden, 01307, Germany
- The Centre for Tactile Internet(CeTI), TU Dresden, Dresden, Germany
| | - Micha Pfeiffer
- Department of Translational Surgical Oncology, National Centre for Tumor Diseases(NCT/UCC), Dresden, 01307, Germany
| | - Fiona Kolbinger
- Department of Translational Surgical Oncology, National Centre for Tumor Diseases(NCT/UCC), Dresden, 01307, Germany
- Department of Visceral, Thoracic and Vascular Surgery, University Hospital and Faculty of Medicine, TU Dresden, 01307, Dresden, Germany
| | - Marius Distler
- Department of Visceral, Thoracic and Vascular Surgery, University Hospital and Faculty of Medicine, TU Dresden, 01307, Dresden, Germany
| | - Jürgen Weitz
- Department of Visceral, Thoracic and Vascular Surgery, University Hospital and Faculty of Medicine, TU Dresden, 01307, Dresden, Germany
- The Centre for Tactile Internet(CeTI), TU Dresden, Dresden, Germany
| | - Stefanie Speidel
- Department of Translational Surgical Oncology, National Centre for Tumor Diseases(NCT/UCC), Dresden, 01307, Germany
- SECAI, TU Dresden, Dresden, Germany
- Department of Visceral, Thoracic and Vascular Surgery, University Hospital and Faculty of Medicine, TU Dresden, 01307, Dresden, Germany
- The Centre for Tactile Internet(CeTI), TU Dresden, Dresden, Germany
| |
Collapse
|
19
|
Wierick A, Schulze A, Bodenstedt S, Speidel S, Distler M, Weitz J, Wagner M. [The digital operating room : Chances and risks of artificial intelligence]. CHIRURGIE (HEIDELBERG, GERMANY) 2024; 95:429-435. [PMID: 38443676 DOI: 10.1007/s00104-024-02058-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Accepted: 02/07/2024] [Indexed: 03/07/2024]
Abstract
At the central workplace of the surgeon the digitalization of the operating room has particular consequences for the surgical work. Starting with intraoperative cross-sectional imaging and sonography, through functional imaging, minimally invasive and robot-assisted surgery up to digital surgical and anesthesiological documentation, the vast majority of operating rooms are now at least partially digitalized. The increasing digitalization of the whole process chain enables not only for the collection but also the analysis of big data. Current research focuses on artificial intelligence for the analysis of intraoperative data as the prerequisite for assistance systems that support surgical decision making or warn of risks; however, these technologies raise new ethical questions for the surgical community that affect the core of surgical work.
Collapse
Affiliation(s)
- Ann Wierick
- Klinik und Poliklinik für Viszeral‑, Thorax- und Gefäßchirurgie, Universitätsklinikum Carl Gustav Carus, Technische Universität Dresden, Fetscherstr. 74, 01307, Dresden, Deutschland
- Nationales Centrum für Tumorerkrankungen (NCT) Dresden, Dresden, Deutschland
| | - André Schulze
- Klinik und Poliklinik für Viszeral‑, Thorax- und Gefäßchirurgie, Universitätsklinikum Carl Gustav Carus, Technische Universität Dresden, Fetscherstr. 74, 01307, Dresden, Deutschland
- Nationales Centrum für Tumorerkrankungen (NCT) Dresden, Dresden, Deutschland
- Zentrum für Taktiles Internet mit Mensch-Maschine-Interaktion (CeTI), Technische Universität Dresden, Dresden, Deutschland
| | - Sebastian Bodenstedt
- Nationales Centrum für Tumorerkrankungen (NCT) Dresden, Dresden, Deutschland
- Zentrum für Taktiles Internet mit Mensch-Maschine-Interaktion (CeTI), Technische Universität Dresden, Dresden, Deutschland
| | - Stefanie Speidel
- Nationales Centrum für Tumorerkrankungen (NCT) Dresden, Dresden, Deutschland
- Zentrum für Taktiles Internet mit Mensch-Maschine-Interaktion (CeTI), Technische Universität Dresden, Dresden, Deutschland
| | - Marius Distler
- Klinik und Poliklinik für Viszeral‑, Thorax- und Gefäßchirurgie, Universitätsklinikum Carl Gustav Carus, Technische Universität Dresden, Fetscherstr. 74, 01307, Dresden, Deutschland
- Nationales Centrum für Tumorerkrankungen (NCT) Dresden, Dresden, Deutschland
- Zentrum für Taktiles Internet mit Mensch-Maschine-Interaktion (CeTI), Technische Universität Dresden, Dresden, Deutschland
| | - Jürgen Weitz
- Klinik und Poliklinik für Viszeral‑, Thorax- und Gefäßchirurgie, Universitätsklinikum Carl Gustav Carus, Technische Universität Dresden, Fetscherstr. 74, 01307, Dresden, Deutschland
- Nationales Centrum für Tumorerkrankungen (NCT) Dresden, Dresden, Deutschland
- Zentrum für Taktiles Internet mit Mensch-Maschine-Interaktion (CeTI), Technische Universität Dresden, Dresden, Deutschland
| | - Martin Wagner
- Klinik und Poliklinik für Viszeral‑, Thorax- und Gefäßchirurgie, Universitätsklinikum Carl Gustav Carus, Technische Universität Dresden, Fetscherstr. 74, 01307, Dresden, Deutschland.
- Nationales Centrum für Tumorerkrankungen (NCT) Dresden, Dresden, Deutschland.
- Zentrum für Taktiles Internet mit Mensch-Maschine-Interaktion (CeTI), Technische Universität Dresden, Dresden, Deutschland.
| |
Collapse
|
20
|
Rivoir D, Funke I, Speidel S. On the pitfalls of Batch Normalization for end-to-end video learning: A study on surgical workflow analysis. Med Image Anal 2024; 94:103126. [PMID: 38452578 DOI: 10.1016/j.media.2024.103126] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/31/2023] [Revised: 01/11/2024] [Accepted: 02/26/2024] [Indexed: 03/09/2024]
Abstract
Batch Normalization's (BN) unique property of depending on other samples in a batch is known to cause problems in several tasks, including sequence modeling. Yet, BN-related issues are hardly studied for long video understanding, despite the ubiquitous use of BN in CNNs (Convolutional Neural Networks) for feature extraction. Especially in surgical workflow analysis, where the lack of pretrained feature extractors has led to complex, multi-stage training pipelines, limited awareness of BN issues may have hidden the benefits of training CNNs and temporal models end to end. In this paper, we analyze pitfalls of BN in video learning, including issues specific to online tasks such as a 'cheating' effect in anticipation. We observe that BN's properties create major obstacles for end-to-end learning. However, using BN-free backbones, even simple CNN-LSTMs beat the state of the art on three surgical workflow benchmarks by utilizing adequate end-to-end training strategies which maximize temporal context. We conclude that awareness of BN's pitfalls is crucial for effective end-to-end learning in surgical tasks. By reproducing results on natural-video datasets, we hope our insights will benefit other areas of video learning as well. Code is available at: https://gitlab.com/nct_tso_public/pitfalls_bn.
Collapse
Affiliation(s)
- Dominik Rivoir
- Department of Translational Surgical Oncology, National Center for Tumor Diseases (NCT/UCC Dresden), Fetscherstraße 74, 01307 Dresden, Germany: German Cancer Research Center (DKFZ), Heidelberg, Germany; Faculty of Medicine and University Hospital Carl Gustav Carus, TUD Dresden University of Technology, Dresden, Germany; Helmholtz-Zentrum Dresden-Rossendorf (HZDR), Dresden, Germany; Centre for Tactile Internet with Human-in-the-Loop (CeTI), TUD Dresden University of Technology, Dresden, Germany.
| | - Isabel Funke
- Department of Translational Surgical Oncology, National Center for Tumor Diseases (NCT/UCC Dresden), Fetscherstraße 74, 01307 Dresden, Germany: German Cancer Research Center (DKFZ), Heidelberg, Germany; Faculty of Medicine and University Hospital Carl Gustav Carus, TUD Dresden University of Technology, Dresden, Germany; Helmholtz-Zentrum Dresden-Rossendorf (HZDR), Dresden, Germany; Centre for Tactile Internet with Human-in-the-Loop (CeTI), TUD Dresden University of Technology, Dresden, Germany
| | - Stefanie Speidel
- Department of Translational Surgical Oncology, National Center for Tumor Diseases (NCT/UCC Dresden), Fetscherstraße 74, 01307 Dresden, Germany: German Cancer Research Center (DKFZ), Heidelberg, Germany; Faculty of Medicine and University Hospital Carl Gustav Carus, TUD Dresden University of Technology, Dresden, Germany; Helmholtz-Zentrum Dresden-Rossendorf (HZDR), Dresden, Germany; Centre for Tactile Internet with Human-in-the-Loop (CeTI), TUD Dresden University of Technology, Dresden, Germany
| |
Collapse
|
21
|
Özsoy E, Czempiel T, Örnek EP, Eck U, Tombari F, Navab N. Holistic OR domain modeling: a semantic scene graph approach. Int J Comput Assist Radiol Surg 2024; 19:791-799. [PMID: 37823976 PMCID: PMC11098880 DOI: 10.1007/s11548-023-03022-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/31/2023] [Accepted: 09/12/2023] [Indexed: 10/13/2023]
Abstract
PURPOSE Surgical procedures take place in highly complex operating rooms (OR), involving medical staff, patients, devices and their interactions. Until now, only medical professionals are capable of comprehending these intricate links and interactions. This work advances the field toward automated, comprehensive and semantic understanding and modeling of the OR domain by introducing semantic scene graphs (SSG) as a novel approach to describing and summarizing surgical environments in a structured and semantically rich manner. METHODS We create the first open-source 4D SSG dataset. 4D-OR includes simulated total knee replacement surgeries captured by RGB-D sensors in a realistic OR simulation center. It includes annotations for SSGs, human and object pose, clinical roles and surgical phase labels. We introduce a neural network-based SSG generation pipeline for semantic reasoning in the OR and apply our approach to two downstream tasks: clinical role prediction and surgical phase recognition. RESULTS We show that our pipeline can successfully reason within the OR domain. The capabilities of our scene graphs are further highlighted by their successful application to clinical role prediction and surgical phase recognition tasks. CONCLUSION This work paves the way for multimodal holistic operating room modeling, with the potential to significantly enhance the state of the art in surgical data analysis, such as enabling more efficient and precise decision-making during surgical procedures, and ultimately improving patient safety and surgical outcomes. We release our code and dataset at github.com/egeozsoy/4D-OR.
Collapse
Affiliation(s)
- Ege Özsoy
- Computer Aided Medical Procedures, Technische Universität München, Garching, Germany.
| | - Tobias Czempiel
- Computer Aided Medical Procedures, Technische Universität München, Garching, Germany
| | - Evin Pınar Örnek
- Computer Aided Medical Procedures, Technische Universität München, Garching, Germany
| | - Ulrich Eck
- Computer Aided Medical Procedures, Technische Universität München, Garching, Germany
| | - Federico Tombari
- Computer Aided Medical Procedures, Technische Universität München, Garching, Germany
- Google, Zurich, Switzerland
| | - Nassir Navab
- Computer Aided Medical Procedures, Technische Universität München, Garching, Germany
| |
Collapse
|
22
|
Varghese C, Harrison EM, O'Grady G, Topol EJ. Artificial intelligence in surgery. Nat Med 2024; 30:1257-1268. [PMID: 38740998 DOI: 10.1038/s41591-024-02970-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/24/2024] [Accepted: 04/03/2024] [Indexed: 05/16/2024]
Abstract
Artificial intelligence (AI) is rapidly emerging in healthcare, yet applications in surgery remain relatively nascent. Here we review the integration of AI in the field of surgery, centering our discussion on multifaceted improvements in surgical care in the preoperative, intraoperative and postoperative space. The emergence of foundation model architectures, wearable technologies and improving surgical data infrastructures is enabling rapid advances in AI interventions and utility. We discuss how maturing AI methods hold the potential to improve patient outcomes, facilitate surgical education and optimize surgical care. We review the current applications of deep learning approaches and outline a vision for future advances through multimodal foundation models.
Collapse
Affiliation(s)
- Chris Varghese
- Department of Surgery, University of Auckland, Auckland, New Zealand
| | - Ewen M Harrison
- Centre for Medical Informatics, Usher Institute, University of Edinburgh, Edinburgh, UK
| | - Greg O'Grady
- Department of Surgery, University of Auckland, Auckland, New Zealand
- Auckland Bioengineering Institute, University of Auckland, Auckland, New Zealand
| | - Eric J Topol
- Scripps Research Translational Institute, La Jolla, CA, USA.
| |
Collapse
|
23
|
Yiu A, Lam K, Simister C, Clarke J, Kinross J. Adoption of routine surgical video recording: a nationwide freedom of information act request across England and Wales. EClinicalMedicine 2024; 70:102545. [PMID: 38685926 PMCID: PMC11056472 DOI: 10.1016/j.eclinm.2024.102545] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/30/2024] [Revised: 02/28/2024] [Accepted: 02/28/2024] [Indexed: 05/02/2024] Open
Abstract
Background Surgical video contains data with significant potential to improve surgical outcome assessment, quality assurance, education, and research. Current utilisation of surgical video recording is unknown and related policies/governance structures are unclear. Methods A nationwide Freedom of Information (FOI) request concerning surgical video recording, technology, consent, access, and governance was sent to all acute National Health Service (NHS) trusts/boards in England/Wales between 20th February and 20th March 2023. Findings 140/144 (97.2%) trusts/boards in England/Wales responded to the FOI request. Surgical procedures were routinely recorded in 22 trusts/boards. The median estimate of consultant surgeons routinely recording their procedures was 20%. Surgical video was stored on internal systems (n = 27), third-party products (n = 29), and both (n = 9). 32/140 (22.9%) trusts/boards ask for consent to record procedures as part of routine care. Consent for recording included non-clinical purposes in 55/140 (39.3%) trusts/boards. Policies for surgeon/patient access to surgical video were available in 48/140 (34.3%) and 32/140 (22.9%) trusts/boards, respectively. Surgical video was used for non-clinical purposes in 64/140 (45.7%) trusts/boards. Governance policies covering surgical video recording, use, and/or storage were available from 59/140 (42.1%) trusts/boards. Interpretation There is significant heterogeneity in surgical video recording practices in England and Wales. A minority of trusts/boards routinely record surgical procedures, with large variation in recording/storage practices indicating scope for NHS-wide coordination. Revision of surgical video consent, accessibility, and governance policies should be prioritised by trusts/boards to protect key stakeholders. Increased availability of surgical video is essential for patients and surgeons to maximally benefit from the ongoing digital transformation of surgery. Funding KL is supported by an NIHR Academic Clinical Fellowship and acknowledges infrastructure support for this research from the National Institute for Health Research (NIHR) Imperial Biomedical Research Centre (BRC).
Collapse
Affiliation(s)
- Andrew Yiu
- Department of Surgery and Cancer, Imperial College London, UK
| | - Kyle Lam
- Department of Surgery and Cancer, Imperial College London, UK
| | | | - Jonathan Clarke
- Department of Surgery and Cancer, Imperial College London, UK
| | - James Kinross
- Department of Surgery and Cancer, Imperial College London, UK
| |
Collapse
|
24
|
Gui S, Wang Z, Chen J, Zhou X, Zhang C, Cao Y. MT4MTL-KD: A Multi-Teacher Knowledge Distillation Framework for Triplet Recognition. IEEE TRANSACTIONS ON MEDICAL IMAGING 2024; 43:1628-1639. [PMID: 38127608 DOI: 10.1109/tmi.2023.3345736] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/23/2023]
Abstract
The recognition of surgical triplets plays a critical role in the practical application of surgical videos. It involves the sub-tasks of recognizing instruments, verbs, and targets, while establishing precise associations between them. Existing methods face two significant challenges in triplet recognition: 1) the imbalanced class distribution of surgical triplets may lead to spurious task association learning, and 2) the feature extractors cannot reconcile local and global context modeling. To overcome these challenges, this paper presents a novel multi-teacher knowledge distillation framework for multi-task triplet learning, known as MT4MTL-KD. MT4MTL-KD leverages teacher models trained on less imbalanced sub-tasks to assist multi-task student learning for triplet recognition. Moreover, we adopt different categories of backbones for the teacher and student models, facilitating the integration of local and global context modeling. To further align the semantic knowledge between the triplet task and its sub-tasks, we propose a novel feature attention module (FAM). This module utilizes attention mechanisms to assign multi-task features to specific sub-tasks. We evaluate the performance of MT4MTL-KD on both the 5-fold cross-validation and the CholecTriplet challenge splits of the CholecT45 dataset. The experimental results consistently demonstrate the superiority of our framework over state-of-the-art methods, achieving significant improvements of up to 6.4% on the cross-validation split.
Collapse
|
25
|
Liu Y, Hayashi Y, Oda M, Kitasaka T, Mori K. YOLOv7-RepFPN: Improving real-time performance of laparoscopic tool detection on embedded systems. Healthc Technol Lett 2024; 11:157-166. [PMID: 38638498 PMCID: PMC11022232 DOI: 10.1049/htl2.12072] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/04/2023] [Accepted: 12/09/2023] [Indexed: 04/20/2024] Open
Abstract
This study focuses on enhancing the inference speed of laparoscopic tool detection on embedded devices. Laparoscopy, a minimally invasive surgery technique, markedly reduces patient recovery times and postoperative complications. Real-time laparoscopic tool detection helps assisting laparoscopy by providing information for surgical navigation, and its implementation on embedded devices is gaining interest due to the portability, network independence and scalability of the devices. However, embedded devices often face computation resource limitations, potentially hindering inference speed. To mitigate this concern, the work introduces a two-fold modification to the YOLOv7 model: the feature channels and integrate RepBlock is halved, yielding the YOLOv7-RepFPN model. This configuration leads to a significant reduction in computational complexity. Additionally, the focal EIoU (efficient intersection of union) loss function is employed for bounding box regression. Experimental results on an embedded device demonstrate that for frame-by-frame laparoscopic tool detection, the proposed YOLOv7-RepFPN achieved an mAP of 88.2% (with IoU set to 0.5) on a custom dataset based on EndoVis17, and an inference speed of 62.9 FPS. Contrasting with the original YOLOv7, which garnered an 89.3% mAP and 41.8 FPS under identical conditions, the methodology enhances the speed by 21.1 FPS while maintaining detection accuracy. This emphasizes the effectiveness of the work.
Collapse
Affiliation(s)
- Yuzhang Liu
- Graduate School of InformaticsNagoya UniversityAichi, NagoyaJapan
| | - Yuichiro Hayashi
- Graduate School of InformaticsNagoya UniversityAichi, NagoyaJapan
| | - Masahiro Oda
- Graduate School of InformaticsNagoya UniversityAichi, NagoyaJapan
- Information and CommunicationsNagoya UniversityAichi NagoyaJapan
| | - Takayuki Kitasaka
- Department of Information ScienceAichi Institute of TechnologyAichi, NagoyaJapan
| | - Kensaku Mori
- Graduate School of InformaticsNagoya UniversityAichi, NagoyaJapan
- Information and CommunicationsNagoya UniversityAichi NagoyaJapan
- Research Center of Medical BigdataNational Institute of InformaticsTokyoJapan
| |
Collapse
|
26
|
Zheng Z, Hayashi Y, Oda M, Kitasaka T, Mori K. Revisiting instrument segmentation: Learning from decentralized surgical sequences with various imperfect annotations. Healthc Technol Lett 2024; 11:146-156. [PMID: 38638500 PMCID: PMC11022234 DOI: 10.1049/htl2.12068] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/02/2023] [Accepted: 12/07/2023] [Indexed: 04/20/2024] Open
Abstract
This paper focuses on a new and challenging problem related to instrument segmentation. This paper aims to learn a generalizable model from distributed datasets with various imperfect annotations. Collecting a large-scale dataset for centralized learning is usually impeded due to data silos and privacy issues. Besides, local clients, such as hospitals or medical institutes, may hold datasets with diverse and imperfect annotations. These datasets can include scarce annotations (many samples are unlabelled), noisy labels prone to errors, and scribble annotations with less precision. Federated learning (FL) has emerged as an attractive paradigm for developing global models with these locally distributed datasets. However, its potential in instrument segmentation has yet to be fully investigated. Moreover, the problem of learning from various imperfect annotations in an FL setup is rarely studied, even though it presents a more practical and beneficial scenario. This work rethinks instrument segmentation in such a setting and propose a practical FL framework for this issue. Notably, this approach surpassed centralized learning under various imperfect annotation settings. This method established a foundational benchmark, and future work can build upon it by considering each client owning various annotations and aligning closer with real-world complexities.
Collapse
Affiliation(s)
- Zhou Zheng
- Graduate School of InformaticsNagoya UniversityChikusa‐ku, NagoyaAichiJapan
| | - Yuichiro Hayashi
- Graduate School of InformaticsNagoya UniversityChikusa‐ku, NagoyaAichiJapan
| | - Masahiro Oda
- Graduate School of InformaticsNagoya UniversityChikusa‐ku, NagoyaAichiJapan
- Information Strategy Office, Information and CommunicationsNagoya UniversityChikusa‐ku, NagoyaAichiJapan
| | - Takayuki Kitasaka
- School of Information ScienceAichi Institute of TechnologyYagusa‐cho, ToyotaAichiJapan
| | - Kensaku Mori
- Graduate School of InformaticsNagoya UniversityChikusa‐ku, NagoyaAichiJapan
- Information Strategy Office, Information and CommunicationsNagoya UniversityChikusa‐ku, NagoyaAichiJapan
- Research Center for Medical BigdataNational Institute of InformaticsChiyoda‐ku, TokyoJapan
| |
Collapse
|
27
|
Lemke HU, Mathis-Ullrich F. Design criteria for AI-based IT systems. Int J Comput Assist Radiol Surg 2024; 19:185-190. [PMID: 38270812 DOI: 10.1007/s11548-024-03064-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 01/08/2024] [Indexed: 01/26/2024]
Abstract
PURPOSE This editorial relates to a panel discussion during the CARS 2023 congress that addressed the question on how AI-based IT systems should be designed that record and (transparently) display a reproducible path on clinical decision making. Even though the software engineering approach suggested for this endeavor is of a generic nature, it is assumed that the listed design criteria are applicable to IT system development also for the domain of radiology and surgery. METHODS An example of a possible design approach is outlined by illustrating on how to move from data, information, knowledge and models to wisdom-based decision making in the context of a conceptual GPT system design. In all these design steps, the essential requirements for system quality, information quality, and service quality may be realized by following the design cycle as suggested by A.R. Hevner, appropriately applied to AI-based IT systems design. RESULTS It can be observed that certain state-of-the-art AI algorithms and systems, such as large language models or generative pre-trained transformers (GPTs), are becoming increasingly complex and, therefore, need to be rigorously examined to render them transparent and comprehensible in their usage for all stakeholders involved in health care. Further critical questions that need to be addressed are outlined and complemented with some suggestions, that a possible design framework for a stakeholder specific AI system could be a (modest) GPT based on a small language model. DISCUSSION A fundamental question for the future remains whether society wants a quasi-wisdom-oriented healthcare system, based on data-driven intelligence with AI, or a human curated wisdom based on model-driven intelligence (with and without AI). Special CARS workshops and think tanks are planned to address this challenging question and possible new direction for assisting selected medical disciplines, e.g., radiology and surgery.
Collapse
Affiliation(s)
- Heinz U Lemke
- International Foundation for Computer Assisted Radiology and Surgery - IFCARS, Küssaberg, Germany.
| | | |
Collapse
|
28
|
Kostiuchik G, Sharan L, Mayer B, Wolf I, Preim B, Engelhardt S. Surgical phase and instrument recognition: how to identify appropriate dataset splits. Int J Comput Assist Radiol Surg 2024:10.1007/s11548-024-03063-9. [PMID: 38285380 DOI: 10.1007/s11548-024-03063-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/01/2023] [Accepted: 01/08/2024] [Indexed: 01/30/2024]
Abstract
PURPOSE Machine learning approaches can only be reliably evaluated if training, validation, and test data splits are representative and not affected by the absence of classes. Surgical workflow and instrument recognition are two tasks that are complicated in this manner, because of heavy data imbalances resulting from different length of phases and their potential erratic occurrences. Furthermore, sub-properties like instrument (co-)occurrence are usually not particularly considered when defining the split. METHODS We present a publicly available data visualization tool that enables interactive exploration of dataset partitions for surgical phase and instrument recognition. The application focuses on the visualization of the occurrence of phases, phase transitions, instruments, and instrument combinations across sets. Particularly, it facilitates assessment of dataset splits, especially regarding identification of sub-optimal dataset splits. RESULTS We performed analysis of the datasets Cholec80, CATARACTS, CaDIS, M2CAI-workflow, and M2CAI-tool using the proposed application. We were able to uncover phase transitions, individual instruments, and combinations of surgical instruments that were not represented in one of the sets. Addressing these issues, we identify possible improvements in the splits using our tool. A user study with ten participants demonstrated that the participants were able to successfully solve a selection of data exploration tasks. CONCLUSION In highly unbalanced class distributions, special care should be taken with respect to the selection of an appropriate dataset split because it can greatly influence the assessments of machine learning approaches. Our interactive tool allows for determination of better splits to improve current practices in the field. The live application is available at https://cardio-ai.github.io/endovis-ml/ .
Collapse
Affiliation(s)
- Georgii Kostiuchik
- Department of Cardiac Surgery, Heidelberg University Hospital, Heidelberg, Germany.
- DZHK (German Centre for Cardiovascular Research), Partner Site Heidelberg/Mannheim, Heidelberg, Germany.
| | - Lalith Sharan
- Department of Cardiac Surgery, Heidelberg University Hospital, Heidelberg, Germany
- DZHK (German Centre for Cardiovascular Research), Partner Site Heidelberg/Mannheim, Heidelberg, Germany
| | - Benedikt Mayer
- Department of Simulation and Graphics, University of Magdeburg, Magdeburg, Germany
| | - Ivo Wolf
- Department of Computer Science, Mannheim University of Applied Sciences, Mannheim, Germany
| | - Bernhard Preim
- Department of Simulation and Graphics, University of Magdeburg, Magdeburg, Germany
| | - Sandy Engelhardt
- Department of Cardiac Surgery, Heidelberg University Hospital, Heidelberg, Germany
- DZHK (German Centre for Cardiovascular Research), Partner Site Heidelberg/Mannheim, Heidelberg, Germany
| |
Collapse
|
29
|
Manoli E, Higginson J, Tolley N, Darzi A, Kinross J, Temelkuran B, Takats Z. Human robotic surgery with intraoperative tissue identification using rapid evaporation ionisation mass spectrometry. Sci Rep 2024; 14:1027. [PMID: 38200062 PMCID: PMC10781715 DOI: 10.1038/s41598-023-50942-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/15/2022] [Accepted: 12/28/2023] [Indexed: 01/12/2024] Open
Abstract
Instantaneous, continuous, and reliable information on the molecular biology of surgical target tissue could significantly contribute to the precision, safety, and speed of the intervention. In this work, we introduced a methodology for chemical tissue identification in robotic surgery using rapid evaporative ionisation mass spectrometry. We developed a surgical aerosol evacuation system that is compatible with a robotic platform enabling consistent intraoperative sample collection and assessed the feasibility of this platform during head and neck surgical cases, using two different surgical energy devices. Our data showed specific, characteristic lipid profiles associated with the tissue type including various ceramides, glycerophospholipids, and glycerolipids, as well as different ion formation mechanisms based on the energy device used. This platform allows continuous and accurate intraoperative mass spectrometry-based identification of ablated/resected tissue and in combination with robotic registration of images, time, and anatomical positions can improve the current robot-assisted surgical platforms and guide surgical strategy.
Collapse
Affiliation(s)
- Eftychios Manoli
- Department of Metabolism, Digestion and Reproduction, Imperial College London, London, UK
| | - James Higginson
- Department of Metabolism, Digestion and Reproduction, Imperial College London, London, UK
| | - Neil Tolley
- Department of Surgery and Cancer, Imperial College London, London, UK
| | - Ara Darzi
- Department of Surgery and Cancer, Imperial College London, London, UK
| | - James Kinross
- Department of Surgery and Cancer, Imperial College London, London, UK
| | - Burak Temelkuran
- Department of Metabolism, Digestion and Reproduction, Imperial College London, London, UK
- The Hamlyn Centre for Robotic Surgery, Imperial College London, London, UK
| | - Zoltan Takats
- Department of Metabolism, Digestion and Reproduction, Imperial College London, London, UK.
- Laboratoire Protéomique, Réponse Inflammatoire et Spectrométrie de Masse (PRISM), Univ. Lille, INSERM U1192, Lille, France.
| |
Collapse
|
30
|
Men Y, Zhao Z, Chen W, Wu H, Zhang G, Luo F, Yu M. Research on workflow recognition for liver rupture repair surgery. MATHEMATICAL BIOSCIENCES AND ENGINEERING : MBE 2024; 21:1844-1856. [PMID: 38454663 DOI: 10.3934/mbe.2024080] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 03/09/2024]
Abstract
Liver rupture repair surgery serves as one tool to treat liver rupture, especially beneficial for cases of mild liver rupture hemorrhage. Liver rupture can catalyze critical conditions such as hemorrhage and shock. Surgical workflow recognition in liver rupture repair surgery videos presents a significant task aimed at reducing surgical mistakes and enhancing the quality of surgeries conducted by surgeons. A liver rupture repair simulation surgical dataset is proposed in this paper which consists of 45 videos collaboratively completed by nine surgeons. Furthermore, an end-to-end SA-RLNet, a self attention-based recurrent convolutional neural network, is introduced in this paper. The self-attention mechanism is used to automatically identify the importance of input features in various instances and associate the relationships between input features. The accuracy of the surgical phase classification of the SA-RLNet approach is 90.6%. The present study demonstrates that the SA-RLNet approach shows strong generalization capabilities on the dataset. SA-RLNet has proved to be advantageous in capturing subtle variations between surgical phases. The application of surgical workflow recognition has promising feasibility in liver rupture repair surgery.
Collapse
Affiliation(s)
- Yutao Men
- Tianjin Key Laboratory for Advanced Mechatronic System Design and Intelligent Control, School of Mechanical Engineering, Tianjin University of Technology, Tianjin 300384, China
- National Demonstration Center for Experimental Mechanical and Electrical Engineering Education, Tianjin University of Technology, Tianjin 300384, China
| | - Zixian Zhao
- Tianjin Key Laboratory for Advanced Mechatronic System Design and Intelligent Control, School of Mechanical Engineering, Tianjin University of Technology, Tianjin 300384, China
- Medical Support Technology Research Department, Systems Engineering Institute, Academy of Military Sciences, People's Liberation Army, Tianjin 300161, China
- National Demonstration Center for Experimental Mechanical and Electrical Engineering Education, Tianjin University of Technology, Tianjin 300384, China
| | - Wei Chen
- Tianjin Key Laboratory for Advanced Mechatronic System Design and Intelligent Control, School of Mechanical Engineering, Tianjin University of Technology, Tianjin 300384, China
- National Demonstration Center for Experimental Mechanical and Electrical Engineering Education, Tianjin University of Technology, Tianjin 300384, China
| | - Hang Wu
- Medical Support Technology Research Department, Systems Engineering Institute, Academy of Military Sciences, People's Liberation Army, Tianjin 300161, China
| | - Guang Zhang
- Medical Support Technology Research Department, Systems Engineering Institute, Academy of Military Sciences, People's Liberation Army, Tianjin 300161, China
| | - Feng Luo
- Medical Support Technology Research Department, Systems Engineering Institute, Academy of Military Sciences, People's Liberation Army, Tianjin 300161, China
| | - Ming Yu
- Medical Support Technology Research Department, Systems Engineering Institute, Academy of Military Sciences, People's Liberation Army, Tianjin 300161, China
| |
Collapse
|
31
|
Mascagni P, Alapatt D, Lapergola A, Vardazaryan A, Mazellier JP, Dallemagne B, Mutter D, Padoy N. Early-stage clinical evaluation of real-time artificial intelligence assistance for laparoscopic cholecystectomy. Br J Surg 2024; 111:znad353. [PMID: 37935636 DOI: 10.1093/bjs/znad353] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/08/2023] [Revised: 07/24/2023] [Accepted: 08/26/2023] [Indexed: 11/09/2023]
Abstract
Lay Summary
The growing availability of surgical digital data and developments in analytics such as artificial intelligence (AI) are being harnessed to improve surgical care. However, technical and cultural barriers to real-time intraoperative AI assistance exist. This early-stage clinical evaluation shows the technical feasibility of concurrently deploying several AIs in operating rooms for real-time assistance during procedures. In addition, potentially relevant clinical applications of these AI models are explored with a multidisciplinary cohort of key stakeholders.
Collapse
Affiliation(s)
- Pietro Mascagni
- ICube, University of Strasbourg, CNRS, IHU Strasbourg, Strasbourg, France
- Department of Medical and Abdominal Surgery and Endocrine-Metabolic Science, Fondazione Policlinico Universitario A. Gemelli IRCCS, Rome, Italy
| | - Deepak Alapatt
- ICube, University of Strasbourg, CNRS, IHU Strasbourg, Strasbourg, France
| | - Alfonso Lapergola
- Department of Digestive and Endocrine Surgery, Nouvel Hôpital Civil, Hôpitaux Universitaires de Strasbourg, Strasbourg, France
| | | | | | - Bernard Dallemagne
- Institute for Research against Digestive Cancer (IRCAD), Strasbourg, France
| | - Didier Mutter
- Department of Digestive and Endocrine Surgery, Nouvel Hôpital Civil, Hôpitaux Universitaires de Strasbourg, Strasbourg, France
- Institute of Image-Guided Surgery, IHU-Strasbourg, Strasbourg, France
| | - Nicolas Padoy
- ICube, University of Strasbourg, CNRS, IHU Strasbourg, Strasbourg, France
- Institute of Image-Guided Surgery, IHU-Strasbourg, Strasbourg, France
| |
Collapse
|
32
|
Tollefson MK, Ross CJ. Defining the Standard for Surgical Video Deidentification. JAMA Surg 2024; 159:104-105. [PMID: 37878296 DOI: 10.1001/jamasurg.2023.1800] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/26/2023]
Abstract
This article reviews the implementation of standards for surgical video deidentification.
Collapse
|
33
|
Zhang J, Barbarisi S, Kadkhodamohammadi A, Stoyanov D, Luengo I. Self-knowledge distillation for surgical phase recognition. Int J Comput Assist Radiol Surg 2024; 19:61-68. [PMID: 37340283 DOI: 10.1007/s11548-023-02970-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/06/2023] [Accepted: 05/19/2023] [Indexed: 06/22/2023]
Abstract
PURPOSE Advances in surgical phase recognition are generally led by training deeper networks. Rather than going further with a more complex solution, we believe that current models can be exploited better. We propose a self-knowledge distillation framework that can be integrated into current state-of-the-art (SOTA) models without requiring any extra complexity to the models or annotations. METHODS Knowledge distillation is a framework for network regularization where knowledge is distilled from a teacher network to a student network. In self-knowledge distillation, the student model becomes the teacher such that the network learns from itself. Most phase recognition models follow an encoder-decoder framework. Our framework utilizes self-knowledge distillation in both stages. The teacher model guides the training process of the student model to extract enhanced feature representations from the encoder and build a more robust temporal decoder to tackle the over-segmentation problem. RESULTS We validate our proposed framework on the public dataset Cholec80. Our framework is embedded on top of four popular SOTA approaches and consistently improves their performance. Specifically, our best GRU model boosts performance by [Formula: see text] accuracy and [Formula: see text] F1-score over the same baseline model. CONCLUSION We embed a self-knowledge distillation framework for the first time in the surgical phase recognition training pipeline. Experimental results demonstrate that our simple yet powerful framework can improve performance of existing phase recognition models. Moreover, our extensive experiments show that even with 75% of the training set we still achieve performance on par with the same baseline model trained on the full set.
Collapse
Affiliation(s)
- Jinglu Zhang
- Medtronic Digital Surgery, 230 City Road, London, UK
| | | | | | - Danail Stoyanov
- Medtronic Digital Surgery, 230 City Road, London, UK
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences, University College London, London, UK
| | - Imanol Luengo
- Medtronic Digital Surgery, 230 City Road, London, UK
| |
Collapse
|
34
|
De Backer P, Peraire Lores M, Demuynck M, Piramide F, Simoens J, Oosterlinck T, Bogaert W, Shan CV, Van Regemorter K, Wastyn A, Checcucci E, Debbaut C, Van Praet C, Farinha R, De Groote R, Gallagher A, Decaestecker K, Mottrie A. Surgical Phase Duration in Robot-Assisted Partial Nephrectomy: A Surgical Data Science Exploration for Clinical Relevance. Diagnostics (Basel) 2023; 13:3386. [PMID: 37958283 PMCID: PMC10650909 DOI: 10.3390/diagnostics13213386] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/25/2023] [Revised: 10/29/2023] [Accepted: 11/03/2023] [Indexed: 11/15/2023] Open
Abstract
(1) Background: Surgical phases form the basic building blocks for surgical skill assessment, feedback, and teaching. The phase duration itself and its correlation with clinical parameters at diagnosis have not yet been investigated. Novel commercial platforms provide phase indications but have not been assessed for accuracy yet. (2) Methods: We assessed 100 robot-assisted partial nephrectomy videos for phase durations based on previously defined proficiency metrics. We developed an annotation framework and subsequently compared our annotations to an existing commercial solution (Touch Surgery, Medtronic™). We subsequently explored clinical correlations between phase durations and parameters derived from diagnosis and treatment. (3) Results: An objective and uniform phase assessment requires precise definitions derived from an iterative revision process. A comparison to a commercial solution shows large differences in definitions across phases. BMI and the duration of renal tumor identification are positively correlated, as are tumor complexity and both tumor excision and renorrhaphy duration. (4) Conclusions: The surgical phase duration can be correlated with certain clinical outcomes. Further research should investigate whether the retrieved correlations are also clinically meaningful. This requires an increase in dataset sizes and facilitation through intelligent computer vision algorithms. Commercial platforms can facilitate this dataset expansion and help unlock the full potential, provided that the phase annotation details are disclosed.
Collapse
Affiliation(s)
- Pieter De Backer
- ORSI Academy, 9090 Melle, Belgium
- IbiTech-Biommeda, Department of Electronics and Information Systems, Faculty of Engineering and Architecture, Ghent University, 9000 Ghent, Belgium
- Department of Human Structure and Repair, Faculty of Medicine and Health Sciences, Ghent University, 9000 Ghent, Belgium (C.V.P.)
- Young Academic Urologist—Urotechnology Working Group, NL-6803 Arnhem, The Netherlands
- Department of Urology, ERN eUROGEN Accredited Centre, Ghent University Hospital, 9000 Ghent, Belgium
| | | | - Meret Demuynck
- Department of Human Structure and Repair, Faculty of Medicine and Health Sciences, Ghent University, 9000 Ghent, Belgium (C.V.P.)
| | - Federico Piramide
- ORSI Academy, 9090 Melle, Belgium
- Department of Surgery, Candiolo Cancer Institute, FPO-IRCCS, 10060 Turin, Italy
| | | | | | - Wouter Bogaert
- IbiTech-Biommeda, Department of Electronics and Information Systems, Faculty of Engineering and Architecture, Ghent University, 9000 Ghent, Belgium
| | - Chi Victor Shan
- Department of Human Structure and Repair, Faculty of Medicine and Health Sciences, Ghent University, 9000 Ghent, Belgium (C.V.P.)
| | - Karel Van Regemorter
- Department of Human Structure and Repair, Faculty of Medicine and Health Sciences, Ghent University, 9000 Ghent, Belgium (C.V.P.)
| | - Aube Wastyn
- Department of Human Structure and Repair, Faculty of Medicine and Health Sciences, Ghent University, 9000 Ghent, Belgium (C.V.P.)
| | - Enrico Checcucci
- Young Academic Urologist—Urotechnology Working Group, NL-6803 Arnhem, The Netherlands
- Department of Surgery, Candiolo Cancer Institute, FPO-IRCCS, 10060 Turin, Italy
| | - Charlotte Debbaut
- IbiTech-Biommeda, Department of Electronics and Information Systems, Faculty of Engineering and Architecture, Ghent University, 9000 Ghent, Belgium
| | - Charles Van Praet
- Department of Human Structure and Repair, Faculty of Medicine and Health Sciences, Ghent University, 9000 Ghent, Belgium (C.V.P.)
- Department of Urology, ERN eUROGEN Accredited Centre, Ghent University Hospital, 9000 Ghent, Belgium
| | | | - Ruben De Groote
- Department of Urology, Onze-Lieve Vrouwziekenhuis Hospital, 9300 Aalst, Belgium
| | | | - Karel Decaestecker
- Department of Human Structure and Repair, Faculty of Medicine and Health Sciences, Ghent University, 9000 Ghent, Belgium (C.V.P.)
- Department of Urology, ERN eUROGEN Accredited Centre, Ghent University Hospital, 9000 Ghent, Belgium
- Department of Urology, AZ Maria Middelares Hospital, 9000 Ghent, Belgium
| | - Alexandre Mottrie
- ORSI Academy, 9090 Melle, Belgium
- Department of Urology, Onze-Lieve Vrouwziekenhuis Hospital, 9300 Aalst, Belgium
| |
Collapse
|
35
|
Grezenko H, Alsadoun L, Farrukh A, Rehman A, Shehryar A, Nathaniel E, Affaf M, I Kh Almadhoun MK, Quinn M. From Nanobots to Neural Networks: Multifaceted Revolution of Artificial Intelligence in Surgical Medicine and Therapeutics. Cureus 2023; 15:e49082. [PMID: 38125253 PMCID: PMC10731389 DOI: 10.7759/cureus.49082] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 11/19/2023] [Indexed: 12/23/2023] Open
Abstract
This comprehensive exploration unveils the transformative potential of Artificial Intelligence (AI) within medicine and surgery. Through a meticulous journey, we examine AI's current applications in healthcare, including medical diagnostics, surgical procedures, and advanced therapeutics. Delving into the theoretical foundations of AI, encompassing machine learning, deep learning, and Natural Language Processing (NLP), we illuminate the critical underpinnings supporting AI's integration into healthcare. Highlighting the symbiotic relationship between humans and machines, we emphasize how AI augments clinical capabilities without supplanting the irreplaceable human touch in healthcare delivery. Also, we'd like to briefly mention critical findings and takeaways they can expect to encounter in the article. A thoughtful analysis of the economic, societal, and ethical implications of AI's integration into healthcare underscores our commitment to addressing critical issues, such as data privacy, algorithmic transparency, and equitable access to AI-driven healthcare services. As we contemplate the future landscape, we project an exciting vista where more sophisticated AI algorithms and real-time surgical visualizations redefine the boundaries of medical achievement. While acknowledging the limitations of the present research, we shed light on AI's pivotal role in enhancing patient engagement, education, and data security within the burgeoning realm of AI-driven healthcare.
Collapse
Affiliation(s)
- Han Grezenko
- Translational Neuroscience, Barrow Neurological Institute, Phoenix, USA
| | - Lara Alsadoun
- Plastic Surgery, Chelsea and Westminster Hospital, London, GBR
| | - Ayesha Farrukh
- Family Medicine, Rawalpindi Medical University, Rawalpindi, PAK
| | | | | | | | - Maryam Affaf
- Internal Medicine, Women's Medical and Dental College, Abbotabad, PAK
| | | | - Maria Quinn
- Internal Medicine, Jinnah Hospital, Lahore, PAK
| |
Collapse
|
36
|
Park B, Chi H, Park B, Lee J, Jin HS, Park S, Hyung WJ, Choi MK. Visual modalities-based multimodal fusion for surgical phase recognition. Comput Biol Med 2023; 166:107453. [PMID: 37774560 DOI: 10.1016/j.compbiomed.2023.107453] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/14/2023] [Revised: 08/17/2023] [Accepted: 09/04/2023] [Indexed: 10/01/2023]
Abstract
Surgical workflow analysis is essential to help optimize surgery by encouraging efficient communication and the use of resources. However, the performance of phase recognition is limited by the use of information related to the presence of surgical instruments. To address the problem, we propose visual modality-based multimodal fusion for surgical phase recognition to overcome the limited diversity of information such as the presence of instruments. Using the proposed methods, we extracted a visual kinematics-based index related to using instruments, such as movement and their interrelations during surgery. In addition, we improved recognition performance using an effective convolutional neural network (CNN)-based fusion method for visual features and a visual kinematics-based index (VKI). The visual kinematics-based index improves the understanding of a surgical procedure since information is related to instrument interaction. Furthermore, these indices can be extracted in any environment, such as laparoscopic surgery, and help obtain complementary information for system kinematics log errors. The proposed methodology was applied to two multimodal datasets, a virtual reality (VR) simulator-based dataset (PETRAW) and a private distal gastrectomy surgery dataset, to verify that it can help improve recognition performance in clinical environments. We also explored the influence of a visual kinematics-based index to recognize each surgical workflow by the instrument's existence and the instrument's trajectory. Through the experimental results of a distal gastrectomy video dataset, we validated the effectiveness of our proposed fusion approach in surgical phase recognition. The relatively simple yet index-incorporated fusion we propose can yield significant performance improvements over only CNN-based training and exhibits effective training results compared to fusion based on Transformers, which require a large amount of pre-trained data.
Collapse
Affiliation(s)
- Bogyu Park
- AI Dev. Group, Hutom, Dokmak-ro 279, Mapo-gu, 04151, Seoul, Republic of Korea.
| | - Hyeongyu Chi
- AI Dev. Group, Hutom, Dokmak-ro 279, Mapo-gu, 04151, Seoul, Republic of Korea.
| | - Bokyung Park
- AI Dev. Group, Hutom, Dokmak-ro 279, Mapo-gu, 04151, Seoul, Republic of Korea.
| | - Jiwon Lee
- AI Dev. Group, Hutom, Dokmak-ro 279, Mapo-gu, 04151, Seoul, Republic of Korea.
| | - Hye Su Jin
- AI Dev. Group, Hutom, Dokmak-ro 279, Mapo-gu, 04151, Seoul, Republic of Korea.
| | - Sunghyun Park
- Yonsei University College of Medicine, Yonsei-ro 50, Seodaemun-gu, 03722, Seoul, Republic of Korea.
| | - Woo Jin Hyung
- AI Dev. Group, Hutom, Dokmak-ro 279, Mapo-gu, 04151, Seoul, Republic of Korea; Yonsei University College of Medicine, Yonsei-ro 50, Seodaemun-gu, 03722, Seoul, Republic of Korea.
| | - Min-Kook Choi
- AI Dev. Group, Hutom, Dokmak-ro 279, Mapo-gu, 04151, Seoul, Republic of Korea.
| |
Collapse
|
37
|
Demir KC, Schieber H, Weise T, Roth D, May M, Maier A, Yang SH. Deep Learning in Surgical Workflow Analysis: A Review of Phase and Step Recognition. IEEE J Biomed Health Inform 2023; 27:5405-5417. [PMID: 37665700 DOI: 10.1109/jbhi.2023.3311628] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 09/06/2023]
Abstract
OBJECTIVE In the last two decades, there has been a growing interest in exploring surgical procedures with statistical models to analyze operations at different semantic levels. This information is necessary for developing context-aware intelligent systems, which can assist the physicians during operations, evaluate procedures afterward or help the management team to effectively utilize the operating room. The objective is to extract reliable patterns from surgical data for the robust estimation of surgical activities performed during operations. The purpose of this article is to review the state-of-the-art deep learning methods that have been published after 2018 for analyzing surgical workflows, with a focus on phase and step recognition. METHODS Three databases, IEEE Xplore, Scopus, and PubMed were searched, and additional studies are added through a manual search. After the database search, 343 studies were screened and a total of 44 studies are selected for this review. CONCLUSION The use of temporal information is essential for identifying the next surgical action. Contemporary methods used mainly RNNs, hierarchical CNNs, and Transformers to preserve long-distance temporal relations. The lack of large publicly available datasets for various procedures is a great challenge for the development of new and robust models. As supervised learning strategies are used to show proof-of-concept, self-supervised, semi-supervised, or active learning methods are used to mitigate dependency on annotated data. SIGNIFICANCE The present study provides a comprehensive review of recent methods in surgical workflow analysis, summarizes commonly used architectures, datasets, and discusses challenges.
Collapse
|
38
|
Brandenburg JM, Jenke AC, Stern A, Daum MTJ, Schulze A, Younis R, Petrynowski P, Davitashvili T, Vanat V, Bhasker N, Schneider S, Mündermann L, Reinke A, Kolbinger FR, Jörns V, Fritz-Kebede F, Dugas M, Maier-Hein L, Klotz R, Distler M, Weitz J, Müller-Stich BP, Speidel S, Bodenstedt S, Wagner M. Active learning for extracting surgomic features in robot-assisted minimally invasive esophagectomy: a prospective annotation study. Surg Endosc 2023; 37:8577-8593. [PMID: 37833509 PMCID: PMC10615926 DOI: 10.1007/s00464-023-10447-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/29/2023] [Accepted: 09/02/2023] [Indexed: 10/15/2023]
Abstract
BACKGROUND With Surgomics, we aim for personalized prediction of the patient's surgical outcome using machine-learning (ML) on multimodal intraoperative data to extract surgomic features as surgical process characteristics. As high-quality annotations by medical experts are crucial, but still a bottleneck, we prospectively investigate active learning (AL) to reduce annotation effort and present automatic recognition of surgomic features. METHODS To establish a process for development of surgomic features, ten video-based features related to bleeding, as highly relevant intraoperative complication, were chosen. They comprise the amount of blood and smoke in the surgical field, six instruments, and two anatomic structures. Annotation of selected frames from robot-assisted minimally invasive esophagectomies was performed by at least three independent medical experts. To test whether AL reduces annotation effort, we performed a prospective annotation study comparing AL with equidistant sampling (EQS) for frame selection. Multiple Bayesian ResNet18 architectures were trained on a multicentric dataset, consisting of 22 videos from two centers. RESULTS In total, 14,004 frames were tag annotated. A mean F1-score of 0.75 ± 0.16 was achieved for all features. The highest F1-score was achieved for the instruments (mean 0.80 ± 0.17). This result is also reflected in the inter-rater-agreement (1-rater-kappa > 0.82). Compared to EQS, AL showed better recognition results for the instruments with a significant difference in the McNemar test comparing correctness of predictions. Moreover, in contrast to EQS, AL selected more frames of the four less common instruments (1512 vs. 607 frames) and achieved higher F1-scores for common instruments while requiring less training frames. CONCLUSION We presented ten surgomic features relevant for bleeding events in esophageal surgery automatically extracted from surgical video using ML. AL showed the potential to reduce annotation effort while keeping ML performance high for selected features. The source code and the trained models are published open source.
Collapse
Affiliation(s)
- Johanna M Brandenburg
- Department of General, Visceral and Transplantation Surgery, Heidelberg University Hospital, Heidelberg, Germany
- National Center for Tumor Diseases (NCT), Heidelberg, Germany
| | - Alexander C Jenke
- Department of Translational Surgical Oncology, National Center for Tumor Diseases (NCT/UCC), Dresden, Germany
- German Cancer Research Center (DKFZ), Heidelberg, Germany
- Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany
- Helmholtz-Zentrum Dresden - Rossendorf (HZDR), Dresden, Germany
| | - Antonia Stern
- Corporate Research and Technology, Karl Storz SE & Co KG, Tuttlingen, Germany
| | - Marie T J Daum
- Department of General, Visceral and Transplantation Surgery, Heidelberg University Hospital, Heidelberg, Germany
- National Center for Tumor Diseases (NCT), Heidelberg, Germany
| | - André Schulze
- Department of General, Visceral and Transplantation Surgery, Heidelberg University Hospital, Heidelberg, Germany
- National Center for Tumor Diseases (NCT), Heidelberg, Germany
| | - Rayan Younis
- Department of General, Visceral and Transplantation Surgery, Heidelberg University Hospital, Heidelberg, Germany
- National Center for Tumor Diseases (NCT), Heidelberg, Germany
| | - Philipp Petrynowski
- Department of General, Visceral and Transplantation Surgery, Heidelberg University Hospital, Heidelberg, Germany
| | - Tornike Davitashvili
- Department of General, Visceral and Transplantation Surgery, Heidelberg University Hospital, Heidelberg, Germany
| | - Vincent Vanat
- Department of General, Visceral and Transplantation Surgery, Heidelberg University Hospital, Heidelberg, Germany
| | - Nithya Bhasker
- Department of Translational Surgical Oncology, National Center for Tumor Diseases (NCT/UCC), Dresden, Germany
- German Cancer Research Center (DKFZ), Heidelberg, Germany
- Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany
- Helmholtz-Zentrum Dresden - Rossendorf (HZDR), Dresden, Germany
| | - Sophia Schneider
- Corporate Research and Technology, Karl Storz SE & Co KG, Tuttlingen, Germany
| | - Lars Mündermann
- Corporate Research and Technology, Karl Storz SE & Co KG, Tuttlingen, Germany
| | - Annika Reinke
- Department of Intelligent Medical Systems (IMSY), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Fiona R Kolbinger
- German Cancer Research Center (DKFZ), Heidelberg, Germany
- Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany
- Helmholtz-Zentrum Dresden - Rossendorf (HZDR), Dresden, Germany
- Department of Visceral-, Thoracic and Vascular Surgery, University Hospital Carl Gustav Carus, Technische Universität Dresden, Fetscherstraße 74, 01307, Dresden, Germany
- Else Kröner-Fresenius Center for Digital Health, Technische Universität Dresden, Dresden, Germany
- National Center for Tumor Diseases (NCT/UCC), Dresden, Germany
| | - Vanessa Jörns
- Institute of Medical Informatics, Heidelberg University Hospital, Heidelberg, Germany
| | - Fleur Fritz-Kebede
- Institute of Medical Informatics, Heidelberg University Hospital, Heidelberg, Germany
| | - Martin Dugas
- Institute of Medical Informatics, Heidelberg University Hospital, Heidelberg, Germany
| | - Lena Maier-Hein
- Department of Intelligent Medical Systems (IMSY), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Rosa Klotz
- Department of General, Visceral and Transplantation Surgery, Heidelberg University Hospital, Heidelberg, Germany
- The Study Center of the German Surgical Society (SDGC), Heidelberg University Hospital, Heidelberg, Germany
| | - Marius Distler
- German Cancer Research Center (DKFZ), Heidelberg, Germany
- Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany
- Helmholtz-Zentrum Dresden - Rossendorf (HZDR), Dresden, Germany
- Department of Visceral-, Thoracic and Vascular Surgery, University Hospital Carl Gustav Carus, Technische Universität Dresden, Fetscherstraße 74, 01307, Dresden, Germany
- National Center for Tumor Diseases (NCT/UCC), Dresden, Germany
| | - Jürgen Weitz
- German Cancer Research Center (DKFZ), Heidelberg, Germany
- Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany
- Helmholtz-Zentrum Dresden - Rossendorf (HZDR), Dresden, Germany
- Department of Visceral-, Thoracic and Vascular Surgery, University Hospital Carl Gustav Carus, Technische Universität Dresden, Fetscherstraße 74, 01307, Dresden, Germany
- National Center for Tumor Diseases (NCT/UCC), Dresden, Germany
- Centre for Tactile Internet With Human-in-the-Loop (CeTI), Technische Universität Dresden, 01062, Dresden, Germany
| | - Beat P Müller-Stich
- National Center for Tumor Diseases (NCT), Heidelberg, Germany
- University Center for Gastrointestinal and Liver Diseases, St. Clara Hospital and University Hospital Basel, Basel, Switzerland
| | - Stefanie Speidel
- Department of Translational Surgical Oncology, National Center for Tumor Diseases (NCT/UCC), Dresden, Germany
- German Cancer Research Center (DKFZ), Heidelberg, Germany
- Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany
- Helmholtz-Zentrum Dresden - Rossendorf (HZDR), Dresden, Germany
- Centre for Tactile Internet With Human-in-the-Loop (CeTI), Technische Universität Dresden, 01062, Dresden, Germany
| | - Sebastian Bodenstedt
- Department of Translational Surgical Oncology, National Center for Tumor Diseases (NCT/UCC), Dresden, Germany
- German Cancer Research Center (DKFZ), Heidelberg, Germany
- Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany
- Helmholtz-Zentrum Dresden - Rossendorf (HZDR), Dresden, Germany
- Centre for Tactile Internet With Human-in-the-Loop (CeTI), Technische Universität Dresden, 01062, Dresden, Germany
| | - Martin Wagner
- Department of General, Visceral and Transplantation Surgery, Heidelberg University Hospital, Heidelberg, Germany.
- National Center for Tumor Diseases (NCT), Heidelberg, Germany.
- German Cancer Research Center (DKFZ), Heidelberg, Germany.
- Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany.
- Helmholtz-Zentrum Dresden - Rossendorf (HZDR), Dresden, Germany.
- Department of Visceral-, Thoracic and Vascular Surgery, University Hospital Carl Gustav Carus, Technische Universität Dresden, Fetscherstraße 74, 01307, Dresden, Germany.
- National Center for Tumor Diseases (NCT/UCC), Dresden, Germany.
- Centre for Tactile Internet With Human-in-the-Loop (CeTI), Technische Universität Dresden, 01062, Dresden, Germany.
| |
Collapse
|
39
|
Spence C, Shah OA, Cebula A, Tucker K, Sochart D, Kader D, Asopa V. Machine learning models to predict surgical case duration compared to current industry standards: scoping review. BJS Open 2023; 7:zrad113. [PMID: 37931236 PMCID: PMC10630142 DOI: 10.1093/bjsopen/zrad113] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/25/2023] [Revised: 09/21/2023] [Accepted: 09/21/2023] [Indexed: 11/08/2023] Open
Abstract
BACKGROUND Surgical waiting lists have risen dramatically across the UK as a result of the COVID-19 pandemic. The effective use of operating theatres by optimal scheduling could help mitigate this, but this requires accurate case duration predictions. Current standards for predicting the duration of surgery are inaccurate. Artificial intelligence (AI) offers the potential for greater accuracy in predicting surgical case duration. This study aimed to investigate whether there is evidence to support that AI is more accurate than current industry standards at predicting surgical case duration, with a secondary aim of analysing whether the implementation of the models used produced efficiency savings. METHOD PubMed, Embase, and MEDLINE libraries were searched through to July 2023 to identify appropriate articles. PRISMA extension for scoping reviews and the Arksey and O'Malley framework were followed. Study quality was assessed using a modified version of the reporting guidelines for surgical AI papers by Farrow et al. Algorithm performance was reported using evaluation metrics. RESULTS The search identified 2593 articles: 14 were suitable for inclusion and 13 reported on the accuracy of AI algorithms against industry standards, with seven demonstrating a statistically significant improvement in prediction accuracy (P < 0.05). The larger studies demonstrated the superiority of neural networks over other machine learning techniques. Efficiency savings were identified in a RCT. Significant methodological limitations were identified across most studies. CONCLUSION The studies suggest that machine learning and deep learning models are more accurate at predicting the duration of surgery; however, further research is required to determine the best way to implement this technology.
Collapse
Affiliation(s)
- Christopher Spence
- Academic Surgical Unit, South West London Elective Orthopaedic Centre, Epsom, Surrey, UK
| | - Owais A Shah
- Academic Surgical Unit, South West London Elective Orthopaedic Centre, Epsom, Surrey, UK
| | - Anna Cebula
- Academic Surgical Unit, South West London Elective Orthopaedic Centre, Epsom, Surrey, UK
| | - Keith Tucker
- Academic Surgical Unit, South West London Elective Orthopaedic Centre, Epsom, Surrey, UK
| | - David Sochart
- Academic Surgical Unit, South West London Elective Orthopaedic Centre, Epsom, Surrey, UK
| | - Deiary Kader
- Academic Surgical Unit, South West London Elective Orthopaedic Centre, Epsom, Surrey, UK
| | - Vipin Asopa
- Academic Surgical Unit, South West London Elective Orthopaedic Centre, Epsom, Surrey, UK
| |
Collapse
|
40
|
Cao J, Yip HC, Chen Y, Scheppach M, Luo X, Yang H, Cheng MK, Long Y, Jin Y, Chiu PWY, Yam Y, Meng HML, Dou Q. Intelligent surgical workflow recognition for endoscopic submucosal dissection with real-time animal study. Nat Commun 2023; 14:6676. [PMID: 37865629 PMCID: PMC10590425 DOI: 10.1038/s41467-023-42451-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/20/2023] [Accepted: 10/11/2023] [Indexed: 10/23/2023] Open
Abstract
Recent advancements in artificial intelligence have witnessed human-level performance; however, AI-enabled cognitive assistance for therapeutic procedures has not been fully explored nor pre-clinically validated. Here we propose AI-Endo, an intelligent surgical workflow recognition suit, for endoscopic submucosal dissection (ESD). Our AI-Endo is trained on high-quality ESD cases from an expert endoscopist, covering a decade time expansion and consisting of 201,026 labeled frames. The learned model demonstrates outstanding performance on validation data, including cases from relatively junior endoscopists with various skill levels, procedures conducted with different endoscopy systems and therapeutic skills, and cohorts from international multi-centers. Furthermore, we integrate our AI-Endo with the Olympus endoscopic system and validate the AI-enabled cognitive assistance system with animal studies in live ESD training sessions. Dedicated data analysis from surgical phase recognition results is summarized in an automatically generated report for skill assessment.
Collapse
Affiliation(s)
- Jianfeng Cao
- Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China
| | - Hon-Chi Yip
- Department of Surgery, The Chinese University of Hong Kong, Hong Kong, China.
| | - Yueyao Chen
- Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China
| | - Markus Scheppach
- Internal Medicine III-Gastroenterology, University Hospital of Augsburg, Augsburg, Germany
| | - Xiaobei Luo
- Guangdong Provincial Key Laboratory of Gastroenterology, Nanfang Hospital, Southern Medical University, Guangzhou, China
| | - Hongzheng Yang
- Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China
| | - Ming Kit Cheng
- Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong, Hong Kong, China
| | - Yonghao Long
- Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China
| | - Yueming Jin
- Department of Biomedical Engineering, National University of Singapore, Singapore, Singapore
| | - Philip Wai-Yan Chiu
- Multi-scale Medical Robotics Center and The Chinese University of Hong Kong, Hong Kong, China.
| | - Yeung Yam
- Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong, Hong Kong, China.
- Multi-scale Medical Robotics Center and The Chinese University of Hong Kong, Hong Kong, China.
- Centre for Perceptual and Interactive Intelligence and The Chinese University of Hong Kong, Hong Kong, China.
| | - Helen Mei-Ling Meng
- Centre for Perceptual and Interactive Intelligence and The Chinese University of Hong Kong, Hong Kong, China.
| | - Qi Dou
- Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China.
| |
Collapse
|
41
|
Ramesh S, Dall'Alba D, Gonzalez C, Yu T, Mascagni P, Mutter D, Marescaux J, Fiorini P, Padoy N. Weakly Supervised Temporal Convolutional Networks for Fine-Grained Surgical Activity Recognition. IEEE TRANSACTIONS ON MEDICAL IMAGING 2023; 42:2592-2602. [PMID: 37030859 DOI: 10.1109/tmi.2023.3262847] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/19/2023]
Abstract
Automatic recognition of fine-grained surgical activities, called steps, is a challenging but crucial task for intelligent intra-operative computer assistance. The development of current vision-based activity recognition methods relies heavily on a high volume of manually annotated data. This data is difficult and time-consuming to generate and requires domain-specific knowledge. In this work, we propose to use coarser and easier-to-annotate activity labels, namely phases, as weak supervision to learn step recognition with fewer step annotated videos. We introduce a step-phase dependency loss to exploit the weak supervision signal. We then employ a Single-Stage Temporal Convolutional Network (SS-TCN) with a ResNet-50 backbone, trained in an end-to-end fashion from weakly annotated videos, for temporal activity segmentation and recognition. We extensively evaluate and show the effectiveness of the proposed method on a large video dataset consisting of 40 laparoscopic gastric bypass procedures and the public benchmark CATARACTS containing 50 cataract surgeries.
Collapse
|
42
|
Liu Z, Hitchcock DB, Singapogu RB. Cannulation Skill Assessment Using Functional Data Analysis. IEEE J Biomed Health Inform 2023; 27:4512-4523. [PMID: 37310836 PMCID: PMC10519736 DOI: 10.1109/jbhi.2023.3283188] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
OBJECTIVE A clinician's operative skill-the ability to safely and effectively perform a procedure-directly impacts patient outcomes and well-being. Therefore, it is necessary to accurately assess skill progression during medical training as well as develop methods to most efficiently train healthcare professionals. METHODS In this study, we explore whether time-series needle angle data recorded during cannulation on a simulator can be analyzed using functional data analysis methods to (1) identify skilled versus unskilled performance and (2) relate angle profiles to degree of success of the procedure. RESULTS Our methods successfully differentiated between types of needle angle profiles. In addition, the identified profile types were associated with degrees of skilled and unskilled behavior of subjects. Furthermore, the types of variability in the dataset were analyzed, providing particular insight into the overall range of needle angles used as well as the rate of change of angle as cannulation progressed in time. Finally, cannulation angle profiles also demonstrated an observable correlation with degree of cannulation success, a metric that is closely related to clinical outcome. CONCLUSION In summary, the methods presented here enable rich assessment of clinical skill since the functional (i.e., dynamic) nature of the data is duly considered.
Collapse
|
43
|
Ramesh S, Dall'Alba D, Gonzalez C, Yu T, Mascagni P, Mutter D, Marescaux J, Fiorini P, Padoy N. TRandAugment: temporal random augmentation strategy for surgical activity recognition from videos. Int J Comput Assist Radiol Surg 2023; 18:1665-1672. [PMID: 36944845 PMCID: PMC10491694 DOI: 10.1007/s11548-023-02864-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/05/2023] [Accepted: 03/01/2023] [Indexed: 03/23/2023]
Abstract
PURPOSE Automatic recognition of surgical activities from intraoperative surgical videos is crucial for developing intelligent support systems for computer-assisted interventions. Current state-of-the-art recognition methods are based on deep learning where data augmentation has shown the potential to improve the generalization of these methods. This has spurred work on automated and simplified augmentation strategies for image classification and object detection on datasets of still images. Extending such augmentation methods to videos is not straightforward, as the temporal dimension needs to be considered. Furthermore, surgical videos pose additional challenges as they are composed of multiple, interconnected, and long-duration activities. METHODS This work proposes a new simplified augmentation method, called TRandAugment, specifically designed for long surgical videos, that treats each video as an assemble of temporal segments and applies consistent but random transformations to each segment. The proposed augmentation method is used to train an end-to-end spatiotemporal model consisting of a CNN (ResNet50) followed by a TCN. RESULTS The effectiveness of the proposed method is demonstrated on two surgical video datasets, namely Bypass40 and CATARACTS, and two tasks, surgical phase and step recognition. TRandAugment adds a performance boost of 1-6% over previous state-of-the-art methods, that uses manually designed augmentations. CONCLUSION This work presents a simplified and automated augmentation method for long surgical videos. The proposed method has been validated on different datasets and tasks indicating the importance of devising temporal augmentation methods for long surgical videos.
Collapse
Affiliation(s)
- Sanat Ramesh
- Altair Robotics Lab, University of Verona, 37134, Verona, Italy.
- ICube, University of Strasbourg, CNRS, 67000, Strasbourg, France.
| | - Diego Dall'Alba
- Altair Robotics Lab, University of Verona, 37134, Verona, Italy
| | - Cristians Gonzalez
- University Hospital of Strasbourg, 67000, Strasbourg, France
- Institute of Image-Guided Surgery, IHU Strasbourg, 67000, Strasbourg, France
| | - Tong Yu
- ICube, University of Strasbourg, CNRS, 67000, Strasbourg, France
| | - Pietro Mascagni
- Institute of Image-Guided Surgery, IHU Strasbourg, 67000, Strasbourg, France
- Fondazione Policlinico Universitario Agostino Gemelli IRCCS, 00168, Rome, Italy
| | - Didier Mutter
- University Hospital of Strasbourg, 67000, Strasbourg, France
- IRCAD, 67000, Strasbourg, France
- Institute of Image-Guided Surgery, IHU Strasbourg, 67000, Strasbourg, France
| | | | - Paolo Fiorini
- Altair Robotics Lab, University of Verona, 37134, Verona, Italy
| | - Nicolas Padoy
- ICube, University of Strasbourg, CNRS, 67000, Strasbourg, France
- Institute of Image-Guided Surgery, IHU Strasbourg, 67000, Strasbourg, France
| |
Collapse
|
44
|
Arabian H, Abdulbaki Alshirbaji T, Jalal NA, Krueger-Ziolek S, Moeller K. P-CSEM: An Attention Module for Improved Laparoscopic Surgical Tool Detection. SENSORS (BASEL, SWITZERLAND) 2023; 23:7257. [PMID: 37631791 PMCID: PMC10459566 DOI: 10.3390/s23167257] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/28/2023] [Revised: 08/14/2023] [Accepted: 08/15/2023] [Indexed: 08/27/2023]
Abstract
Minimal invasive surgery, more specifically laparoscopic surgery, is an active topic in the field of research. The collaboration between surgeons and new technologies aims to improve operation procedures as well as to ensure the safety of patients. An integral part of operating rooms modernization is the real-time communication between the surgeon and the data gathered using the numerous devices during surgery. A fundamental tool that can aid surgeons during laparoscopic surgery is the recognition of the different phases during an operation. Current research has shown a correlation between the surgical tools utilized and the present phase of surgery. To this end, a robust surgical tool classifier is desired for optimal performance. In this paper, a deep learning framework embedded with a custom attention module, the P-CSEM, has been proposed to refine the spatial features for surgical tool classification in laparoscopic surgery videos. This approach utilizes convolutional neural networks (CNNs) integrated with P-CSEM attention modules at different levels of the architecture for improved feature refinement. The model was trained and tested on the popular, publicly available Cholec80 database. Results showed that the attention integrated model achieved a mean average precision of 93.14%, and visualizations revealed the ability of the model to adhere more towards features of tool relevance. The proposed approach displays the benefits of integrating attention modules into surgical tool classification models for a more robust and precise detection.
Collapse
Affiliation(s)
- Herag Arabian
- Institute of Technical Medicine (ITeM), Furtwangen University, 78054 Villingen-Schwenningen, Germany
| | - Tamer Abdulbaki Alshirbaji
- Institute of Technical Medicine (ITeM), Furtwangen University, 78054 Villingen-Schwenningen, Germany
- Innovation Center Computer Assisted Surgery (ICCAS), University of Leipzig, 04103 Leipzig, Germany
| | - Nour Aldeen Jalal
- Institute of Technical Medicine (ITeM), Furtwangen University, 78054 Villingen-Schwenningen, Germany
- Innovation Center Computer Assisted Surgery (ICCAS), University of Leipzig, 04103 Leipzig, Germany
| | - Sabine Krueger-Ziolek
- Institute of Technical Medicine (ITeM), Furtwangen University, 78054 Villingen-Schwenningen, Germany
| | - Knut Moeller
- Institute of Technical Medicine (ITeM), Furtwangen University, 78054 Villingen-Schwenningen, Germany
- Department of Mechanical Engineering, University of Canterbury, Christchurch 8041, New Zealand
- Department of Microsystems Engineering, University of Freiburg, 79110 Freiburg, Germany
| |
Collapse
|
45
|
Tranter-Entwistle I, Simcock C, Eglinton T, Connor S. Prospective cohort study of operative outcomes in laparoscopic cholecystectomy using operative difficulty grade-adjusted CUSUM analysis. Br J Surg 2023; 110:1068-1071. [PMID: 36882185 PMCID: PMC10416680 DOI: 10.1093/bjs/znad046] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/13/2022] [Revised: 01/19/2023] [Accepted: 02/07/2023] [Indexed: 03/09/2023]
Affiliation(s)
| | - Corin Simcock
- Department of Surgery, The University of Otago Medical School, Christchurch, New Zealand
| | - Tim Eglinton
- Department of Surgery, The University of Otago Medical School, Christchurch, New Zealand
- Department of General Surgery, Christchurch Hospital, CDHB, Christchurch, New Zealand
| | - Saxon Connor
- Department of General Surgery, Christchurch Hospital, CDHB, Christchurch, New Zealand
| |
Collapse
|
46
|
Henn J, Hatterscheidt S, Sahu A, Buness A, Dohmen J, Arensmeyer J, Feodorovici P, Sommer N, Schmidt J, Kalff JC, Matthaei H. Machine Learning for Decision-Support in Acute Abdominal Pain - Proof of Concept and Central Considerations. Zentralbl Chir 2023; 148:376-383. [PMID: 37562397 DOI: 10.1055/a-2125-1559] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 08/12/2023]
Abstract
Acute abdominal pain is a common presenting symptom in the emergency department and represents heterogeneous causes and diagnoses. There is often a decision to be made regarding emergency surgical care. Machine learning (ML) could be used here as a decision-support and relieve the time and personnel resource shortage.Patients with acute abdominal pain presenting to the Department of Surgery at Bonn University Hospital in 2020 and 2021 were retrospectively analyzed. Clinical parameters as well as laboratory values were used as predictors. After randomly splitting into a training and test data set (ratio 80 to 20), three ML algorithms were comparatively trained and validated. The entire procedure was repeated 20 times.A total of 1357 patients were identified and included in the analysis, with one in five (n = 276, 20.3%) requiring emergency abdominal surgery within 24 hours. Patients operated on were more likely to be male (p = 0.026), older (p = 0.006), had more gastrointestinal symptoms (nausea: p < 0.001, vomiting p < 0.001) as well as a more recent onset of pain (p < 0.001). Tenderness (p < 0.001) and guarding (p < 0.001) were more common in surgically treated patients and blood analyses showed increased inflammation levels (white blood cell count: p < 0.001, CRP: p < 0.001) and onset of organ dysfunction (creatinine: p < 0.014, quick p < 0.001). Of the three trained algorithms, the tree-based methods (h2o random forest and cforest) showed the best performance. The algorithms classified patients, i.e., predicted surgery, with a median AUC ROC of 0.81 and 0.79 and AUC PRC of 0.56 in test sets.A proof-of-concept was achieved with the development of an ML model for predicting timely surgical therapy for acute abdomen. The ML algorithm can be a valuable tool in decision-making. Especially in the context of heavily used medical resources, the algorithm can help to use these scarce resources more effectively. Technological progress, especially regarding artificial intelligence, increasingly enables evidence-based approaches in surgery but requires a strictly interdisciplinary approach. In the future, the use and handling of ML should be integrated into surgical training.
Collapse
Affiliation(s)
- Jonas Henn
- Department of General, Visceral, Thoracic and Vascular Surgery, University Hospital Bonn, Bonn, Germany
| | - Simon Hatterscheidt
- Department of General, Visceral, Thoracic and Vascular Surgery, University Hospital Bonn, Bonn, Germany
| | - Anshupa Sahu
- Institute for Medical Biometry, Informatics and Epidemiology, University Hospital Bonn, Bonn, Germany
- Institute for Genomic Statistics and Bioinformatics, University Hospital Bonn, Bonn, Germany
| | - Andreas Buness
- Institute for Medical Biometry, Informatics and Epidemiology, University Hospital Bonn, Bonn, Germany
- Institute for Genomic Statistics and Bioinformatics, University Hospital Bonn, Bonn, Germany
| | - Jonas Dohmen
- Department of General, Visceral, Thoracic and Vascular Surgery, University Hospital Bonn, Bonn, Germany
| | - Jan Arensmeyer
- Division of Thoracic Surgery, Department of General, Visceral, Thoracic and Vascular Surgery, University Hospital Bonn, Bonn, Germany
| | - Philipp Feodorovici
- Division of Thoracic Surgery, Department of General, Visceral, Thoracic and Vascular Surgery, University Hospital Bonn, Bonn, Germany
| | - Nils Sommer
- Department of General, Visceral, Thoracic and Vascular Surgery, University Hospital Bonn, Bonn, Germany
| | - Joachim Schmidt
- Division of Thoracic Surgery, Department of General, Visceral, Thoracic and Vascular Surgery, University Hospital Bonn, Bonn, Germany
- Department of Thoracic Surgery, Helios Hospital Bonn Rhein-Sieg, Bonn, Germany
| | - Jörg C Kalff
- Department of General, Visceral, Thoracic and Vascular Surgery, University Hospital Bonn, Bonn, Germany
| | - Hanno Matthaei
- Department of General, Visceral, Thoracic and Vascular Surgery, University Hospital Bonn, Bonn, Germany
| |
Collapse
|
47
|
Kolbinger FR, Bodenstedt S, Carstens M, Leger S, Krell S, Rinner FM, Nielen TP, Kirchberg J, Fritzmann J, Weitz J, Distler M, Speidel S. Artificial Intelligence for context-aware surgical guidance in complex robot-assisted oncological procedures: An exploratory feasibility study. EUROPEAN JOURNAL OF SURGICAL ONCOLOGY 2023:106996. [PMID: 37591704 DOI: 10.1016/j.ejso.2023.106996] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/13/2023] [Revised: 06/20/2023] [Accepted: 07/26/2023] [Indexed: 08/19/2023]
Abstract
INTRODUCTION Complex oncological procedures pose various surgical challenges including dissection in distinct tissue planes and preservation of vulnerable anatomical structures throughout different surgical phases. In rectal surgery, violation of dissection planes increases the risk of local recurrence and autonomous nerve damage resulting in incontinence and sexual dysfunction. This work explores the feasibility of phase recognition and target structure segmentation in robot-assisted rectal resection (RARR) using machine learning. MATERIALS AND METHODS A total of 57 RARR were recorded and subsets of these were annotated with respect to surgical phases and exact locations of target structures (anatomical structures, tissue types, static structures, and dissection areas). For surgical phase recognition, three machine learning models were trained: LSTM, MSTCN, and Trans-SVNet. Based on pixel-wise annotations of target structures in 9037 images, individual segmentation models based on DeepLabv3 were trained. Model performance was evaluated using F1 score, Intersection-over-Union (IoU), accuracy, precision, recall, and specificity. RESULTS The best results for phase recognition were achieved with the MSTCN model (F1 score: 0.82 ± 0.01, accuracy: 0.84 ± 0.03). Mean IoUs for target structure segmentation ranged from 0.14 ± 0.22 to 0.80 ± 0.14 for organs and tissue types and from 0.11 ± 0.11 to 0.44 ± 0.30 for dissection areas. Image quality, distorting factors (i.e. blood, smoke), and technical challenges (i.e. lack of depth perception) considerably impacted segmentation performance. CONCLUSION Machine learning-based phase recognition and segmentation of selected target structures are feasible in RARR. In the future, such functionalities could be integrated into a context-aware surgical guidance system for rectal surgery.
Collapse
Affiliation(s)
- Fiona R Kolbinger
- Department of Visceral, Thoracic and Vascular Surgery, University Hospital and Faculty of Medicine Carl Gustav Carus, Technische Universität Dresden, Fetscherstraße 74, 01307 Dresden, Germany; National Center for Tumor Diseases Dresden (NCT/UCC), Germany: German Cancer Research Center (DKFZ), Heidelberg, Germany; Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany; Helmholtz-Zentrum Dresden - Rossendorf, Dresden, Germany; Else Kröner Fresenius Center for Digital Health (EKFZ), Technische Universität Dresden, Fetscherstraße 74, 01307, Dresden, Germany.
| | - Sebastian Bodenstedt
- Department of Translational Surgical Oncology, National Center for Tumor Diseases (NCT/UCC), Partner Site Dresden, Fetscherstraße 74, 01307, Dresden, Germany; Centre for Tactile Internet with Human-in-the-Loop (CeTI), Technische Universität Dresden, Dresden, Germany
| | - Matthias Carstens
- Department of Visceral, Thoracic and Vascular Surgery, University Hospital and Faculty of Medicine Carl Gustav Carus, Technische Universität Dresden, Fetscherstraße 74, 01307 Dresden, Germany
| | - Stefan Leger
- Else Kröner Fresenius Center for Digital Health (EKFZ), Technische Universität Dresden, Fetscherstraße 74, 01307, Dresden, Germany; Department of Translational Surgical Oncology, National Center for Tumor Diseases (NCT/UCC), Partner Site Dresden, Fetscherstraße 74, 01307, Dresden, Germany
| | - Stefanie Krell
- Department of Translational Surgical Oncology, National Center for Tumor Diseases (NCT/UCC), Partner Site Dresden, Fetscherstraße 74, 01307, Dresden, Germany
| | - Franziska M Rinner
- Department of Visceral, Thoracic and Vascular Surgery, University Hospital and Faculty of Medicine Carl Gustav Carus, Technische Universität Dresden, Fetscherstraße 74, 01307 Dresden, Germany
| | - Thomas P Nielen
- Department of Visceral, Thoracic and Vascular Surgery, University Hospital and Faculty of Medicine Carl Gustav Carus, Technische Universität Dresden, Fetscherstraße 74, 01307 Dresden, Germany
| | - Johanna Kirchberg
- Department of Visceral, Thoracic and Vascular Surgery, University Hospital and Faculty of Medicine Carl Gustav Carus, Technische Universität Dresden, Fetscherstraße 74, 01307 Dresden, Germany; National Center for Tumor Diseases Dresden (NCT/UCC), Germany: German Cancer Research Center (DKFZ), Heidelberg, Germany; Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany; Helmholtz-Zentrum Dresden - Rossendorf, Dresden, Germany
| | - Johannes Fritzmann
- Department of Visceral, Thoracic and Vascular Surgery, University Hospital and Faculty of Medicine Carl Gustav Carus, Technische Universität Dresden, Fetscherstraße 74, 01307 Dresden, Germany; National Center for Tumor Diseases Dresden (NCT/UCC), Germany: German Cancer Research Center (DKFZ), Heidelberg, Germany; Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany; Helmholtz-Zentrum Dresden - Rossendorf, Dresden, Germany
| | - Jürgen Weitz
- Department of Visceral, Thoracic and Vascular Surgery, University Hospital and Faculty of Medicine Carl Gustav Carus, Technische Universität Dresden, Fetscherstraße 74, 01307 Dresden, Germany; National Center for Tumor Diseases Dresden (NCT/UCC), Germany: German Cancer Research Center (DKFZ), Heidelberg, Germany; Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany; Helmholtz-Zentrum Dresden - Rossendorf, Dresden, Germany; Else Kröner Fresenius Center for Digital Health (EKFZ), Technische Universität Dresden, Fetscherstraße 74, 01307, Dresden, Germany; Centre for Tactile Internet with Human-in-the-Loop (CeTI), Technische Universität Dresden, Dresden, Germany
| | - Marius Distler
- Department of Visceral, Thoracic and Vascular Surgery, University Hospital and Faculty of Medicine Carl Gustav Carus, Technische Universität Dresden, Fetscherstraße 74, 01307 Dresden, Germany; National Center for Tumor Diseases Dresden (NCT/UCC), Germany: German Cancer Research Center (DKFZ), Heidelberg, Germany; Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany; Helmholtz-Zentrum Dresden - Rossendorf, Dresden, Germany
| | - Stefanie Speidel
- Else Kröner Fresenius Center for Digital Health (EKFZ), Technische Universität Dresden, Fetscherstraße 74, 01307, Dresden, Germany; Department of Translational Surgical Oncology, National Center for Tumor Diseases (NCT/UCC), Partner Site Dresden, Fetscherstraße 74, 01307, Dresden, Germany; Centre for Tactile Internet with Human-in-the-Loop (CeTI), Technische Universität Dresden, Dresden, Germany.
| |
Collapse
|
48
|
Pereira-Prado V, Martins-Silveira F, Sicco E, Hochmann J, Isiordia-Espinoza MA, González RG, Pandiar D, Bologna-Molina R. Artificial Intelligence for Image Analysis in Oral Squamous Cell Carcinoma: A Review. Diagnostics (Basel) 2023; 13:2416. [PMID: 37510160 PMCID: PMC10378350 DOI: 10.3390/diagnostics13142416] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/26/2023] [Revised: 07/12/2023] [Accepted: 07/17/2023] [Indexed: 07/30/2023] Open
Abstract
Head and neck tumor differential diagnosis and prognosis have always been a challenge for oral pathologists due to their similarities and complexity. Artificial intelligence novel applications can function as an auxiliary tool for the objective interpretation of histomorphological digital slides. In this review, we present digital histopathological image analysis applications in oral squamous cell carcinoma. A literature search was performed in PubMed MEDLINE with the following keywords: "artificial intelligence" OR "deep learning" OR "machine learning" AND "oral squamous cell carcinoma". Artificial intelligence has proven to be a helpful tool in histopathological image analysis of tumors and other lesions, even though it is necessary to continue researching in this area, mainly for clinical validation.
Collapse
Affiliation(s)
- Vanesa Pereira-Prado
- Molecular Pathology Area, School of Dentistry, Universidad de la República, Montevideo 11400, Uruguay
| | - Felipe Martins-Silveira
- Molecular Pathology Area, School of Dentistry, Universidad de la República, Montevideo 11400, Uruguay
| | - Estafanía Sicco
- Molecular Pathology Area, School of Dentistry, Universidad de la República, Montevideo 11400, Uruguay
| | - Jimena Hochmann
- Molecular Pathology Area, School of Dentistry, Universidad de la República, Montevideo 11400, Uruguay
| | - Mario Alberto Isiordia-Espinoza
- Department of Clinics, Los Altos University Center, Institute of Research in Medical Sciences, University of Guadalajara, Guadalajara 44100, Mexico
| | - Rogelio González González
- Research Department, School of Dentistry, Universidad Juárez del Estado de Durango, Durango 34000, Mexico
| | - Deepak Pandiar
- Department of Oral Pathology and Microbiology, Saveetha Dental College and Hospitals, Chennai 600077, India
| | - Ronell Bologna-Molina
- Molecular Pathology Area, School of Dentistry, Universidad de la República, Montevideo 11400, Uruguay
- Research Department, School of Dentistry, Universidad Juárez del Estado de Durango, Durango 34000, Mexico
| |
Collapse
|
49
|
Baghdadi A, Lama S, Singh R, Sutherland GR. Tool-tissue force segmentation and pattern recognition for evaluating neurosurgical performance. Sci Rep 2023; 13:9591. [PMID: 37311965 DOI: 10.1038/s41598-023-36702-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/16/2022] [Accepted: 06/08/2023] [Indexed: 06/15/2023] Open
Abstract
Surgical data quantification and comprehension expose subtle patterns in tasks and performance. Enabling surgical devices with artificial intelligence provides surgeons with personalized and objective performance evaluation: a virtual surgical assist. Here we present machine learning models developed for analyzing surgical finesse using tool-tissue interaction force data in surgical dissection obtained from a sensorized bipolar forceps. Data modeling was performed using 50 neurosurgery procedures that involved elective surgical treatment for various intracranial pathologies. The data collection was conducted by 13 surgeons of varying experience levels using sensorized bipolar forceps, SmartForceps System. The machine learning algorithm constituted design and implementation for three primary purposes, i.e., force profile segmentation for obtaining active periods of tool utilization using T-U-Net, surgical skill classification into Expert and Novice, and surgical task recognition into two primary categories of Coagulation versus non-Coagulation using FTFIT deep learning architectures. The final report to surgeon was a dashboard containing recognized segments of force application categorized into skill and task classes along with performance metrics charts compared to expert level surgeons. Operating room data recording of > 161 h containing approximately 3.6 K periods of tool operation was utilized. The modeling resulted in Weighted F1-score = 0.95 and AUC = 0.99 for force profile segmentation using T-U-Net, Weighted F1-score = 0.71 and AUC = 0.81 for surgical skill classification, and Weighted F1-score = 0.82 and AUC = 0.89 for surgical task recognition using a subset of hand-crafted features augmented to FTFIT neural network. This study delivers a novel machine learning module in a cloud, enabling an end-to-end platform for intraoperative surgical performance monitoring and evaluation. Accessed through a secure application for professional connectivity, a paradigm for data-driven learning is established.
Collapse
Affiliation(s)
- Amir Baghdadi
- Project neuroArm, Department of Clinical Neurosciences, Hotchkiss Brain Institute University of Calgary, Calgary, AB, Canada
| | - Sanju Lama
- Project neuroArm, Department of Clinical Neurosciences, Hotchkiss Brain Institute University of Calgary, Calgary, AB, Canada
| | - Rahul Singh
- Project neuroArm, Department of Clinical Neurosciences, Hotchkiss Brain Institute University of Calgary, Calgary, AB, Canada
| | - Garnette R Sutherland
- Project neuroArm, Department of Clinical Neurosciences, Hotchkiss Brain Institute University of Calgary, Calgary, AB, Canada.
| |
Collapse
|
50
|
Liu Z, Bible J, Petersen L, Zhang Z, Roy-Chaudhury P, Singapogu R. Relating process and outcome metrics for meaningful and interpretable cannulation skill assessment: A machine learning paradigm. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2023; 236:107429. [PMID: 37119772 PMCID: PMC10291517 DOI: 10.1016/j.cmpb.2023.107429] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/02/2022] [Revised: 02/06/2023] [Accepted: 02/15/2023] [Indexed: 05/21/2023]
Abstract
BACKGROUND AND OBJECTIVES The quality of healthcare delivery depends directly on the skills of clinicians. For patients on hemodialysis, medical errors or injuries caused during cannulation can lead to adverse outcomes, including potential death. To promote objective skill assessment and effective training, we present a machine learning approach, which utilizes a highly-sensorized cannulation simulator and a set of objective process and outcome metrics. METHODS In this study, 52 clinicians were recruited to perform a set of pre-defined cannulation tasks on the simulator. Based on data collected by sensors during their task performance, the feature space was then constructed based on force, motion, and infrared sensor data. Following this, three machine learning models- support vector machine (SVM), support vector regression (SVR), and elastic net (EN)- were constructed to relate the feature space to objective outcome metrics. Our models utilize classification based on the conventional skill classification labels as well as a new method that represents skill on a continuum. RESULTS With less than 5% of trials misplaced by two classes, the SVM model was effective in predicting skill based on the feature space. In addition, the SVR model effectively places both skill and outcome on a fine-grained continuum (versus discrete divisions) that is representative of reality. As importantly, the elastic net model enabled the identification of a set of process metrics that highly impact outcomes of the cannulation task, including smoothness of motion, needle angles, and pinch forces. CONCLUSIONS The proposed cannulation simulator, paired with machine learning assessment, demonstrates definite advantages over current cannulation training practices. The methods presented here can be adopted to drastically increase the effectiveness of skill assessment and training, thereby potentially improving clinical outcomes of hemodialysis treatment.
Collapse
Affiliation(s)
- Zhanhe Liu
- Department of Bioengineering, Clemson University, 301 Rhodes Research Center, Clemson, 29634, SC, USA
| | - Joe Bible
- School of Mathematical and Statistical Sciences, Clemson University, O-110 Martin Hall, Clemson, 29634, SC, USA
| | - Lydia Petersen
- Department of Bioengineering, Clemson University, 301 Rhodes Research Center, Clemson, 29634, SC, USA
| | - Ziyang Zhang
- Department of Bioengineering, Clemson University, 301 Rhodes Research Center, Clemson, 29634, SC, USA
| | - Prabir Roy-Chaudhury
- UNC Kidney Center, University of North Carolina, Chapel Hill, NC, 28144, USA; (Bill Hefner) VA Medical Center, Salisbury, NC, 28144, USA
| | - Ravikiran Singapogu
- Department of Bioengineering, Clemson University, 301 Rhodes Research Center, Clemson, 29634, SC, USA.
| |
Collapse
|