1
|
Yamada Y, Colan J, Davila A, Hasegawa Y. Multimodal semi-supervised learning for online recognition of multi-granularity surgical workflows. Int J Comput Assist Radiol Surg 2024; 19:1075-1083. [PMID: 38558289 PMCID: PMC11178653 DOI: 10.1007/s11548-024-03101-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/29/2024] [Accepted: 03/04/2024] [Indexed: 04/04/2024]
Abstract
Purpose Surgical workflow recognition is a challenging task that requires understanding multiple aspects of surgery, such as gestures, phases, and steps. However, most existing methods focus on single-task or single-modal models and rely on costly annotations for training. To address these limitations, we propose a novel semi-supervised learning approach that leverages multimodal data and self-supervision to create meaningful representations for various surgical tasks. Methods Our representation learning approach conducts two processes. In the first stage, time contrastive learning is used to learn spatiotemporal visual features from video data, without any labels. In the second stage, multimodal VAE fuses the visual features with kinematic data to obtain a shared representation, which is fed into recurrent neural networks for online recognition. Results Our method is evaluated on two datasets: JIGSAWS and MISAW. We confirmed that it achieved comparable or better performance in multi-granularity workflow recognition compared to fully supervised models specialized for each task. On the JIGSAWS Suturing dataset, we achieve a gesture recognition accuracy of 83.3%. In addition, our model is more efficient in annotation usage, as it can maintain high performance with only half of the labels. On the MISAW dataset, we achieve 84.0% AD-Accuracy in phase recognition and 56.8% AD-Accuracy in step recognition. Conclusion Our multimodal representation exhibits versatility across various surgical tasks and enhances annotation efficiency. This work has significant implications for real-time decision-making systems within the operating room.
Collapse
Affiliation(s)
- Yutaro Yamada
- Department of Micro-Nano Mechanical Science and Engineering, Nagoya University, Furo-cho, Chikusa-ku, Nagoya, Aichi, 464-8603, Japan.
| | - Jacinto Colan
- Department of Micro-Nano Mechanical Science and Engineering, Nagoya University, Furo-cho, Chikusa-ku, Nagoya, Aichi, 464-8603, Japan
| | - Ana Davila
- Institutes of Innovation for Future Society, Nagoya University, Furo-cho, Chikusa-ku, Nagoya, Aichi, 464-8601, Japan
| | - Yasuhisa Hasegawa
- Department of Micro-Nano Mechanical Science and Engineering, Nagoya University, Furo-cho, Chikusa-ku, Nagoya, Aichi, 464-8603, Japan
| |
Collapse
|
2
|
Demir KC, Schieber H, Weise T, Roth D, May M, Maier A, Yang SH. Deep Learning in Surgical Workflow Analysis: A Review of Phase and Step Recognition. IEEE J Biomed Health Inform 2023; 27:5405-5417. [PMID: 37665700 DOI: 10.1109/jbhi.2023.3311628] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 09/06/2023]
Abstract
OBJECTIVE In the last two decades, there has been a growing interest in exploring surgical procedures with statistical models to analyze operations at different semantic levels. This information is necessary for developing context-aware intelligent systems, which can assist the physicians during operations, evaluate procedures afterward or help the management team to effectively utilize the operating room. The objective is to extract reliable patterns from surgical data for the robust estimation of surgical activities performed during operations. The purpose of this article is to review the state-of-the-art deep learning methods that have been published after 2018 for analyzing surgical workflows, with a focus on phase and step recognition. METHODS Three databases, IEEE Xplore, Scopus, and PubMed were searched, and additional studies are added through a manual search. After the database search, 343 studies were screened and a total of 44 studies are selected for this review. CONCLUSION The use of temporal information is essential for identifying the next surgical action. Contemporary methods used mainly RNNs, hierarchical CNNs, and Transformers to preserve long-distance temporal relations. The lack of large publicly available datasets for various procedures is a great challenge for the development of new and robust models. As supervised learning strategies are used to show proof-of-concept, self-supervised, semi-supervised, or active learning methods are used to mitigate dependency on annotated data. SIGNIFICANCE The present study provides a comprehensive review of recent methods in surgical workflow analysis, summarizes commonly used architectures, datasets, and discusses challenges.
Collapse
|
3
|
von Schudnat C, Schoeneberg KP, Albors-Garrigos J, Lahmann B, De-Miguel-Molina M. The Economic Impact of Standardization and Digitalization in the Operating Room: A Systematic Literature Review. J Med Syst 2023; 47:55. [PMID: 37129717 DOI: 10.1007/s10916-023-01945-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/31/2022] [Accepted: 03/29/2023] [Indexed: 05/03/2023]
Abstract
Hospital face increased resource constraints and competition. This escalates the need for efficiency optimization especially in resource-intense areas, such as the Operating Room (OR). Efficiency cannot happen at expenses of patient outcomes. Innovative digital support systems (DSS) have been introduced into the market to support established standardization methods of intraoperative workflows further. This review aimed to analyze whether applied standardization methods and implemented DSS of intraoperative surgical workflows lead to increasing efficiency and demonstrate economic improvements. A systematic review of intraoperative surgical workflows standardization and digitalization was performed. Journal articles and reviews from 2000 to 2023 were retrieved from EBSCO, PubMed, and Scopus databases, as well as the internal database of Johnson & Johnson. 17 articles showed a significant increase in efficiency through standardization, which led to cost reductions between $70.20 to $3,516 per case without negatively impacting quality. Five additional articles on DSS demonstrated a significant positive impact on efficiency and quality. Reduction in OR-time between 6 to 22% per case was one main contributor. No literature on DSS revealed any correlated economic impact. Selected standardization methods and introduced DSS for intraoperative surgical workflows effectively increase efficiency while maintaining or even improving quality. Demonstrated cost-effectiveness of non-digital standardization methods across surgical areas requires more research on complex and resource-intensive procedures and the economic value of DSS to support hospital management's strategic decisions to overcome the increasing economic burden.
Collapse
Affiliation(s)
- Christian von Schudnat
- Department of Business Organization, Faculty of Business Management, Universitat Politecnica de Valencia, Cami de Vera, s/n, 46022, Valencia, Spain.
| | - Klaus-Peter Schoeneberg
- Department of Economic and Social Sciences, Berliner Hochschule für Technik, Berlin, Luxemburger Str. 10, 13353, Berlin, Germany
| | - Jose Albors-Garrigos
- Department of Business Organization, Faculty of Business Management, Universitat Politecnica de Valencia, Cami de Vera, s/n, 46022, Valencia, Spain
| | - Benjamin Lahmann
- Department of Statistics and Operation Analysis, Faculty of Business and Economics, Mendel University Brno, Zemědělská 1, 61300, Brno, Czech Republic
| | - María De-Miguel-Molina
- Department of Business Organization, Faculty of Business Management, Universitat Politecnica de Valencia, Cami de Vera, s/n, 46022, Valencia, Spain
| |
Collapse
|
4
|
Neumann J, Uciteli A, Meschke T, Bieck R, Franke S, Herre H, Neumuth T. Ontology-based surgical workflow recognition and prediction. J Biomed Inform 2022; 136:104240. [DOI: 10.1016/j.jbi.2022.104240] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/23/2022] [Revised: 10/27/2022] [Accepted: 11/03/2022] [Indexed: 11/09/2022]
|
5
|
Khan DZ, Luengo I, Barbarisi S, Addis C, Culshaw L, Dorward NL, Haikka P, Jain A, Kerr K, Koh CH, Layard Horsfall H, Muirhead W, Palmisciano P, Vasey B, Stoyanov D, Marcus HJ. Automated operative workflow analysis of endoscopic pituitary surgery using machine learning: development and preclinical evaluation (IDEAL stage 0). J Neurosurg 2022; 137:51-58. [PMID: 34740198 PMCID: PMC10243668 DOI: 10.3171/2021.6.jns21923] [Citation(s) in RCA: 11] [Impact Index Per Article: 5.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/08/2021] [Accepted: 06/15/2021] [Indexed: 11/06/2022]
Abstract
OBJECTIVE Surgical workflow analysis involves systematically breaking down operations into key phases and steps. Automatic analysis of this workflow has potential uses for surgical training, preoperative planning, and outcome prediction. Recent advances in machine learning (ML) and computer vision have allowed accurate automated workflow analysis of operative videos. In this Idea, Development, Exploration, Assessment, Long-term study (IDEAL) stage 0 study, the authors sought to use Touch Surgery for the development and validation of an ML-powered analysis of phases and steps in the endoscopic transsphenoidal approach (eTSA) for pituitary adenoma resection, a first for neurosurgery. METHODS The surgical phases and steps of 50 anonymized eTSA operative videos were labeled by expert surgeons. Forty videos were used to train a combined convolutional and recurrent neural network model by Touch Surgery. Ten videos were used for model evaluation (accuracy, F1 score), comparing the phase and step recognition of surgeons to the automatic detection of the ML model. RESULTS The longest phase was the sellar phase (median 28 minutes), followed by the nasal phase (median 22 minutes) and the closure phase (median 14 minutes). The longest steps were step 5 (tumor identification and excision, median 17 minutes); step 3 (posterior septectomy and removal of sphenoid septations, median 14 minutes); and step 4 (anterior sellar wall removal, median 10 minutes). There were substantial variations within the recorded procedures in terms of video appearances, step duration, and step order, with only 50% of videos containing all 7 steps performed sequentially in numerical order. Despite this, the model was able to output accurate recognition of surgical phases (91% accuracy, 90% F1 score) and steps (76% accuracy, 75% F1 score). CONCLUSIONS In this IDEAL stage 0 study, ML techniques have been developed to automatically analyze operative videos of eTSA pituitary surgery. This technology has previously been shown to be acceptable to neurosurgical teams and patients. ML-based surgical workflow analysis has numerous potential uses-such as education (e.g., automatic indexing of contemporary operative videos for teaching), improved operative efficiency (e.g., orchestrating the entire surgical team to a common workflow), and improved patient outcomes (e.g., comparison of surgical techniques or early detection of adverse events). Future directions include the real-time integration of Touch Surgery into the live operative environment as an IDEAL stage 1 (first-in-human) study, and further development of underpinning ML models using larger data sets.
Collapse
Affiliation(s)
- Danyal Z. Khan
- Department of Neurosurgery, National Hospital for Neurology and Neurosurgery, London
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences, University College London
| | - Imanol Luengo
- Digital Surgery Ltd., Medtronic, London, United Kingdom
| | | | - Carole Addis
- Digital Surgery Ltd., Medtronic, London, United Kingdom
| | - Lucy Culshaw
- Digital Surgery Ltd., Medtronic, London, United Kingdom
| | - Neil L. Dorward
- Department of Neurosurgery, National Hospital for Neurology and Neurosurgery, London
| | - Pinja Haikka
- Digital Surgery Ltd., Medtronic, London, United Kingdom
| | - Abhiney Jain
- Department of Neurosurgery, National Hospital for Neurology and Neurosurgery, London
| | - Karen Kerr
- Digital Surgery Ltd., Medtronic, London, United Kingdom
| | - Chan Hee Koh
- Department of Neurosurgery, National Hospital for Neurology and Neurosurgery, London
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences, University College London
| | - Hugo Layard Horsfall
- Department of Neurosurgery, National Hospital for Neurology and Neurosurgery, London
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences, University College London
| | - William Muirhead
- Department of Neurosurgery, National Hospital for Neurology and Neurosurgery, London
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences, University College London
| | - Paolo Palmisciano
- Department of Neurosurgery, Trauma Center, Gamma Knife Center, Cannizzaro Hospital, Catania, Italy; and
| | - Baptiste Vasey
- Nuffield Department of Surgical Sciences, University of Oxford, United Kingdom
| | - Danail Stoyanov
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences, University College London
- Digital Surgery Ltd., Medtronic, London, United Kingdom
| | - Hani J. Marcus
- Department of Neurosurgery, National Hospital for Neurology and Neurosurgery, London
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences, University College London
| |
Collapse
|
6
|
Jalal NA, Abdulbaki Alshirbaji T, Laufer B, Docherty PD, Russo SG, Neumuth T, Moller K. Effects of Intra-Abdominal Pressure on Lung Mechanics during Laparoscopic Gynaecology . ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2021; 2021:2091-2094. [PMID: 34891701 DOI: 10.1109/embc46164.2021.9630753] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/14/2023]
Abstract
Investigating the relations between surgical actions and physiological reactions of the patient is essential for developing pre-emptive model-based systems. In this study, the effects of insufflating abdominal cavity with CO2 in laparoscopic gynaecology on the respiration system were analysed. Real-time recordings of anaesthesiology and surgical data of five subjects were acquired and processed, and the correlation between lung mechanics and the intra-abdominal pressure was evaluated. Alterations of ventilation settings undertaken by the anaesthesiologist were also considered. Experimental results demonstrated the high correlation with a mean Pearson coefficient of 0.931.Clinical Relevance- This study demonstrates the effects of intra-abdominal pressure during laparoscopy on lung mechanics and enables developing predictive models to promote a greater awareness in operating rooms.
Collapse
|
7
|
Williams S, Layard Horsfall H, Funnell JP, Hanrahan JG, Khan DZ, Muirhead W, Stoyanov D, Marcus HJ. Artificial Intelligence in Brain Tumour Surgery-An Emerging Paradigm. Cancers (Basel) 2021; 13:cancers13195010. [PMID: 34638495 PMCID: PMC8508169 DOI: 10.3390/cancers13195010] [Citation(s) in RCA: 18] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/07/2021] [Revised: 10/02/2021] [Accepted: 10/03/2021] [Indexed: 01/01/2023] Open
Abstract
Artificial intelligence (AI) platforms have the potential to cause a paradigm shift in brain tumour surgery. Brain tumour surgery augmented with AI can result in safer and more effective treatment. In this review article, we explore the current and future role of AI in patients undergoing brain tumour surgery, including aiding diagnosis, optimising the surgical plan, providing support during the operation, and better predicting the prognosis. Finally, we discuss barriers to the successful clinical implementation, the ethical concerns, and we provide our perspective on how the field could be advanced.
Collapse
Affiliation(s)
- Simon Williams
- Department of Neurosurgery, National Hospital for Neurology and Neurosurgery, London WC1N 3BG, UK; (H.L.H.); (J.P.F.); (J.G.H.); (D.Z.K.); (W.M.); (H.J.M.)
- Wellcome/Engineering and Physical Sciences Research Council (EPSRC) Centre for Interventional and Surgical Sciences (WEISS), London W1W 7TY, UK;
- Correspondence:
| | - Hugo Layard Horsfall
- Department of Neurosurgery, National Hospital for Neurology and Neurosurgery, London WC1N 3BG, UK; (H.L.H.); (J.P.F.); (J.G.H.); (D.Z.K.); (W.M.); (H.J.M.)
- Wellcome/Engineering and Physical Sciences Research Council (EPSRC) Centre for Interventional and Surgical Sciences (WEISS), London W1W 7TY, UK;
| | - Jonathan P. Funnell
- Department of Neurosurgery, National Hospital for Neurology and Neurosurgery, London WC1N 3BG, UK; (H.L.H.); (J.P.F.); (J.G.H.); (D.Z.K.); (W.M.); (H.J.M.)
- Wellcome/Engineering and Physical Sciences Research Council (EPSRC) Centre for Interventional and Surgical Sciences (WEISS), London W1W 7TY, UK;
| | - John G. Hanrahan
- Department of Neurosurgery, National Hospital for Neurology and Neurosurgery, London WC1N 3BG, UK; (H.L.H.); (J.P.F.); (J.G.H.); (D.Z.K.); (W.M.); (H.J.M.)
- Wellcome/Engineering and Physical Sciences Research Council (EPSRC) Centre for Interventional and Surgical Sciences (WEISS), London W1W 7TY, UK;
| | - Danyal Z. Khan
- Department of Neurosurgery, National Hospital for Neurology and Neurosurgery, London WC1N 3BG, UK; (H.L.H.); (J.P.F.); (J.G.H.); (D.Z.K.); (W.M.); (H.J.M.)
- Wellcome/Engineering and Physical Sciences Research Council (EPSRC) Centre for Interventional and Surgical Sciences (WEISS), London W1W 7TY, UK;
| | - William Muirhead
- Department of Neurosurgery, National Hospital for Neurology and Neurosurgery, London WC1N 3BG, UK; (H.L.H.); (J.P.F.); (J.G.H.); (D.Z.K.); (W.M.); (H.J.M.)
- Wellcome/Engineering and Physical Sciences Research Council (EPSRC) Centre for Interventional and Surgical Sciences (WEISS), London W1W 7TY, UK;
| | - Danail Stoyanov
- Wellcome/Engineering and Physical Sciences Research Council (EPSRC) Centre for Interventional and Surgical Sciences (WEISS), London W1W 7TY, UK;
| | - Hani J. Marcus
- Department of Neurosurgery, National Hospital for Neurology and Neurosurgery, London WC1N 3BG, UK; (H.L.H.); (J.P.F.); (J.G.H.); (D.Z.K.); (W.M.); (H.J.M.)
- Wellcome/Engineering and Physical Sciences Research Council (EPSRC) Centre for Interventional and Surgical Sciences (WEISS), London W1W 7TY, UK;
| |
Collapse
|
8
|
Maktabi M, Neumuth T. Situation-Dependent Medical Device Risk Estimation: Design and Evaluation of an Equipment Management Center For Vendor-Independent Integrated Operating Rooms. J Patient Saf 2021; 17:e622-e630. [PMID: 29278578 DOI: 10.1097/pts.0000000000000455] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/26/2022]
Abstract
OBJECTIVES The complexity of surgical interventions and the number of technologies involved are constantly rising. Hospital staff has to learn how to handle new medical devices efficiently. However, if medical device-related incidents occur, the patient treatment is delayed. Patient safety could therefore be supported by an optimized assistance system that helps improve the management of technical equipment by nonmedical hospital staff. METHODS We developed a system for the optimal monitoring of networked medical device activity and maintenance requirements, which works in conjunction with a vendor-independent integrated operating room and an accurate surgical intervention Time And Resource Management System. An integrated situation-dependent risk assessment system gives the medical engineers optimal awareness of the medical devices in the operating room. RESULTS A qualitative and quantitative survey among ten medical engineers from three different hospitals was performed to evaluate the approach. A series of 25 questions was used to evaluate various aspects of our system as well as the system currently used. Moreover, the respondents were asked to perform five tasks related to system supervision and incident handling. Our system received a very positive feedback. The evaluation studies showed that the integration of information, the structured presentation of information, and the assistance modules provide valuable support to medical engineers. CONCLUSIONS An automated operating room monitoring system with an integrated risk assessment and Time And Resource Management System module is a new way to assist the staff being outside of a vendor-independent integrated operating room, who are nevertheless involved in processes in the operating room.
Collapse
Affiliation(s)
- Marianne Maktabi
- From the University of Leipzig, Innovation Center Computer Assisted Surgery (ICCAS), Leipzig, Germany
| | | |
Collapse
|
9
|
van Amsterdam B, Clarkson MJ, Stoyanov D. Gesture Recognition in Robotic Surgery: A Review. IEEE Trans Biomed Eng 2021; 68:2021-2035. [PMID: 33497324 DOI: 10.1109/tbme.2021.3054828] [Citation(s) in RCA: 23] [Impact Index Per Article: 7.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/10/2022]
Abstract
OBJECTIVE Surgical activity recognition is a fundamental step in computer-assisted interventions. This paper reviews the state-of-the-art in methods for automatic recognition of fine-grained gestures in robotic surgery focusing on recent data-driven approaches and outlines the open questions and future research directions. METHODS An article search was performed on 5 bibliographic databases with the following search terms: robotic, robot-assisted, JIGSAWS, surgery, surgical, gesture, fine-grained, surgeme, action, trajectory, segmentation, recognition, parsing. Selected articles were classified based on the level of supervision required for training and divided into different groups representing major frameworks for time series analysis and data modelling. RESULTS A total of 52 articles were reviewed. The research field is showing rapid expansion, with the majority of articles published in the last 4 years. Deep-learning-based temporal models with discriminative feature extraction and multi-modal data integration have demonstrated promising results on small surgical datasets. Currently, unsupervised methods perform significantly less well than the supervised approaches. CONCLUSION The development of large and diverse open-source datasets of annotated demonstrations is essential for development and validation of robust solutions for surgical gesture recognition. While new strategies for discriminative feature extraction and knowledge transfer, or unsupervised and semi-supervised approaches, can mitigate the need for data and labels, they have not yet been demonstrated to achieve comparable performance. Important future research directions include detection and forecast of gesture-specific errors and anomalies. SIGNIFICANCE This paper is a comprehensive and structured analysis of surgical gesture recognition methods aiming to summarize the status of this rapidly evolving field.
Collapse
|
10
|
Computer Vision Analysis of Intraoperative Video: Automated Recognition of Operative Steps in Laparoscopic Sleeve Gastrectomy. Ann Surg 2020; 270:414-421. [PMID: 31274652 DOI: 10.1097/sla.0000000000003460] [Citation(s) in RCA: 150] [Impact Index Per Article: 37.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/30/2022]
Abstract
OBJECTIVE(S) To develop and assess AI algorithms to identify operative steps in laparoscopic sleeve gastrectomy (LSG). BACKGROUND Computer vision, a form of artificial intelligence (AI), allows for quantitative analysis of video by computers for identification of objects and patterns, such as in autonomous driving. METHODS Intraoperative video from LSG from an academic institution was annotated by 2 fellowship-trained, board-certified bariatric surgeons. Videos were segmented into the following steps: 1) port placement, 2) liver retraction, 3) liver biopsy, 4) gastrocolic ligament dissection, 5) stapling of the stomach, 6) bagging specimen, and 7) final inspection of staple line. Deep neural networks were used to analyze videos. Accuracy of operative step identification by the AI was determined by comparing to surgeon annotations. RESULTS Eighty-eight cases of LSG were analyzed. A random 70% sample of these clips was used to train the AI and 30% to test the AI's performance. Mean concordance correlation coefficient for human annotators was 0.862, suggesting excellent agreement. Mean (±SD) accuracy of the AI in identifying operative steps in the test set was 82% ± 4% with a maximum of 85.6%. CONCLUSIONS AI can extract quantitative surgical data from video with 85.6% accuracy. This suggests operative video could be used as a quantitative data source for research in intraoperative clinical decision support, risk prediction, or outcomes studies.
Collapse
|
11
|
Twinanda AP, Yengera G, Mutter D, Marescaux J, Padoy N. RSDNet: Learning to Predict Remaining Surgery Duration from Laparoscopic Videos Without Manual Annotations. IEEE TRANSACTIONS ON MEDICAL IMAGING 2019; 38:1069-1078. [PMID: 30371356 DOI: 10.1109/tmi.2018.2878055] [Citation(s) in RCA: 47] [Impact Index Per Article: 9.4] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/25/2023]
Abstract
Accurate surgery duration estimation is necessary for optimal OR planning, which plays an important role in patient comfort and safety as well as resource optimization. It is, however, challenging to preoperatively predict surgery duration since it varies significantly depending on the patient condition, surgeon skills, and intraoperative situation. In this paper, we propose a deep learning pipeline, referred to as RSDNet, which automatically estimates the remaining surgery duration (RSD) intraoperatively by using only visual information from laparoscopic videos. The previous state-of-the-art approaches for RSD prediction are dependent on manual annotation, whose generation requires expensive expert knowledge and is time-consuming, especially considering the numerous types of surgeries performed in a hospital and the large number of laparoscopic videos available. A crucial feature of RSDNet is that it does not depend on any manual annotation during training, making it easily scalable to many kinds of surgeries. The generalizability of our approach is demonstrated by testing the pipeline on two large datasets containing different types of surgeries: 120 cholecystectomy and 170 gastric bypass videos. The experimental results also show that the proposed network significantly outperforms a traditional method of estimating RSD without utilizing manual annotation. Further, this paper provides a deeper insight into the deep learning network through visualization and interpretation of the features that are automatically learned.
Collapse
|
12
|
Spangenberg N, Augenstein C, Wilke M, Franczyk B. An Intelligent and Data-Driven Decision Support Solution for the Online Surgery Scheduling Problem. ENTERP INF SYST-UK 2019. [DOI: 10.1007/978-3-030-26169-6_5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/26/2022]
|
13
|
Abstract
Healthcare in general, and surgery/interventional care in particular, is evolving through rapid advances in technology and increasing complexity of care, with the goal of maximizing the quality and value of care. Whereas innovations in diagnostic and therapeutic technologies have driven past improvements in the quality of surgical care, future transformation in care will be enabled by data. Conventional methodologies, such as registry studies, are limited in their scope for discovery and research, extent and complexity of data, breadth of analytical techniques, and translation or integration of research findings into patient care. We foresee the emergence of surgical/interventional data science (SDS) as a key element to addressing these limitations and creating a sustainable path toward evidence-based improvement of interventional healthcare pathways. SDS will create tools to measure, model, and quantify the pathways or processes within the context of patient health states or outcomes and use information gained to inform healthcare decisions, guidelines, best practices, policy, and training, thereby improving the safety and quality of healthcare and its value. Data are pervasive throughout the surgical care pathway; thus, SDS can impact various aspects of care, including prevention, diagnosis, intervention, or postoperative recovery. The existing literature already provides preliminary results, suggesting how a data science approach to surgical decision-making could more accurately predict severe complications using complex data from preoperative, intraoperative, and postoperative contexts, how it could support intraoperative decision-making using both existing knowledge and continuous data streams throughout the surgical care pathway, and how it could enable effective collaboration between human care providers and intelligent technologies. In addition, SDS is poised to play a central role in surgical education, for example, through objective assessments, automated virtual coaching, and robot-assisted active learning of surgical skill. However, the potential for transforming surgical care and training through SDS may only be realized through a cultural shift that not only institutionalizes technology to seamlessly capture data but also assimilates individuals with expertise in data science into clinical research teams. Furthermore, collaboration with industry partners from the inception of the discovery process promotes optimal design of data products as well as their efficient translation and commercialization. As surgery continues to evolve through advances in technology that enhance delivery of care, SDS represents a new knowledge domain to engineer surgical care of the future.
Collapse
Affiliation(s)
- S Swaroop Vedula
- The Malone Center for Engineering in Healthcare, The Johns Hopkins University, Baltimore, USA
| | - Gregory D Hager
- The Malone Center for Engineering in Healthcare, The Johns Hopkins University, Baltimore, USA
| |
Collapse
|