1
|
Engelhardt S, Dar SUH, Sharan L, André F, Nagel E, Thomas S. Artificial intelligence in cardiovascular imaging and intervention. Herz 2024:10.1007/s00059-024-05264-z. [PMID: 39120735 DOI: 10.1007/s00059-024-05264-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 07/17/2024] [Indexed: 08/10/2024]
Abstract
Recent progress in artificial intelligence (AI) includes generative models, multimodal foundation models, and federated learning, which enable a wide spectrum of novel exciting applications and scenarios for cardiac image analysis and cardiovascular interventions. The disruptive nature of these novel technologies enables concurrent text and image analysis by so-called vision-language transformer models. They not only allow for automatic derivation of image reports, synthesis of novel images conditioned on certain textual properties, and visual questioning and answering in an oral or written dialogue style, but also for the retrieval of medical images from a large database based on a description of the pathology or specifics of the dataset of interest. Federated learning is an additional ingredient in these novel developments, facilitating multi-centric collaborative training of AI approaches and therefore access to large clinical cohorts. In this review paper, we provide an overview of the recent developments in the field of cardiovascular imaging and intervention and offer a future outlook.
Collapse
Affiliation(s)
- Sandy Engelhardt
- Abteilung für Kardiologie, Angiologie und Pneumologie, Universitätsklinikum Heidelberg, Im Neuenheimer Feld 410, 69120, Heidelberg, Germany.
- Standort Heidelberg-Mannheim, DZHK, Deutsches Zentrum für Herz-Kreislauf-Forschung e. V., Heidelberg, Germany.
| | - Salman Ul Hussan Dar
- Abteilung für Kardiologie, Angiologie und Pneumologie, Universitätsklinikum Heidelberg, Im Neuenheimer Feld 410, 69120, Heidelberg, Germany
- Standort Heidelberg-Mannheim, DZHK, Deutsches Zentrum für Herz-Kreislauf-Forschung e. V., Heidelberg, Germany
| | - Lalith Sharan
- Abteilung für Kardiologie, Angiologie und Pneumologie, Universitätsklinikum Heidelberg, Im Neuenheimer Feld 410, 69120, Heidelberg, Germany
- Standort Heidelberg-Mannheim, DZHK, Deutsches Zentrum für Herz-Kreislauf-Forschung e. V., Heidelberg, Germany
| | - Florian André
- Abteilung für Kardiologie, Angiologie und Pneumologie, Universitätsklinikum Heidelberg, Im Neuenheimer Feld 410, 69120, Heidelberg, Germany
- Standort Heidelberg-Mannheim, DZHK, Deutsches Zentrum für Herz-Kreislauf-Forschung e. V., Heidelberg, Germany
| | - Eike Nagel
- Institut für Experimentelle und Translationale Kardiovaskuläre Bildgebung, Universitätsklinikum Frankfurt, Frankfurt am Main, Germany
- Standort Rhein-Main, DZHK, Deutsches Zentrum für Herz-Kreislauf-Forschung e. V., Frankfurt am Main, Germany
| | - Sarina Thomas
- Department of Informatics, University of Oslo, Oslo, Norway
| |
Collapse
|
2
|
Komatsu M, Kitaguchi D, Yura M, Takeshita N, Yoshida M, Yamaguchi M, Kondo H, Kinoshita T, Ito M. Automatic surgical phase recognition-based skill assessment in laparoscopic distal gastrectomy using multicenter videos. Gastric Cancer 2024; 27:187-196. [PMID: 38038811 DOI: 10.1007/s10120-023-01450-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/20/2023] [Accepted: 10/31/2023] [Indexed: 12/02/2023]
Abstract
BACKGROUND Gastric surgery involves numerous surgical phases; however, its steps can be clearly defined. Deep learning-based surgical phase recognition can promote stylization of gastric surgery with applications in automatic surgical skill assessment. This study aimed to develop a deep learning-based surgical phase-recognition model using multicenter videos of laparoscopic distal gastrectomy, and examine the feasibility of automatic surgical skill assessment using the developed model. METHODS Surgical videos from 20 hospitals were used. Laparoscopic distal gastrectomy was defined and annotated into nine phases and a deep learning-based image classification model was developed for phase recognition. We examined whether the developed model's output, including the number of frames in each phase and the adequacy of the surgical field development during the phase of supra-pancreatic lymphadenectomy, correlated with the manually assigned skill assessment score. RESULTS The overall accuracy of phase recognition was 88.8%. Regarding surgical skill assessment based on the number of frames during the phases of lymphadenectomy of the left greater curvature and reconstruction, the number of frames in the high-score group were significantly less than those in the low-score group (829 vs. 1,152, P < 0.01; 1,208 vs. 1,586, P = 0.01, respectively). The output score of the adequacy of the surgical field development, which is the developed model's output, was significantly higher in the high-score group than that in the low-score group (0.975 vs. 0.970, P = 0.04). CONCLUSION The developed model had high accuracy in phase-recognition tasks and has the potential for application in automatic surgical skill assessment systems.
Collapse
Affiliation(s)
- Masaru Komatsu
- Gastric Surgery Division, National Cancer Center Hospital East, 6-5-1 Kashiwanoha, Kashiwa, Chiba, 277-8577, Japan
- Department for the Promotion of Medical Device Innovation, National Cancer Center Hospital East, 6-5-1 Kashiwanoha, Kashiwa, Chiba, 277-8577, Japan
- Course of Advanced Clinical Research of Cancer, Juntendo University Graduate School of Medicine, 2-1-1, Hongo, Bunkyo-Ward, Tokyo, 113-8421, Japan
| | - Daichi Kitaguchi
- Department for the Promotion of Medical Device Innovation, National Cancer Center Hospital East, 6-5-1 Kashiwanoha, Kashiwa, Chiba, 277-8577, Japan
| | - Masahiro Yura
- Gastric Surgery Division, National Cancer Center Hospital East, 6-5-1 Kashiwanoha, Kashiwa, Chiba, 277-8577, Japan
| | - Nobuyoshi Takeshita
- Department for the Promotion of Medical Device Innovation, National Cancer Center Hospital East, 6-5-1 Kashiwanoha, Kashiwa, Chiba, 277-8577, Japan
| | - Mitsumasa Yoshida
- Gastric Surgery Division, National Cancer Center Hospital East, 6-5-1 Kashiwanoha, Kashiwa, Chiba, 277-8577, Japan
| | - Masayuki Yamaguchi
- Course of Advanced Clinical Research of Cancer, Juntendo University Graduate School of Medicine, 2-1-1, Hongo, Bunkyo-Ward, Tokyo, 113-8421, Japan
| | - Hibiki Kondo
- Department for the Promotion of Medical Device Innovation, National Cancer Center Hospital East, 6-5-1 Kashiwanoha, Kashiwa, Chiba, 277-8577, Japan
| | - Takahiro Kinoshita
- Gastric Surgery Division, National Cancer Center Hospital East, 6-5-1 Kashiwanoha, Kashiwa, Chiba, 277-8577, Japan
| | - Masaaki Ito
- Department for the Promotion of Medical Device Innovation, National Cancer Center Hospital East, 6-5-1 Kashiwanoha, Kashiwa, Chiba, 277-8577, Japan.
- Surgical Device Innovation Office, National Cancer Center Hospital East, 6-5-1, Kashiwanoha, Kashiwa, Chiba, 277-8577, Japan.
| |
Collapse
|
3
|
Junger D, Just E, Brandenburg JM, Wagner M, Schaumann K, Klenzner T, Burgert O. Toward an interoperable, intraoperative situation recognition system via process modeling, execution, and control using the standards BPMN and CMMN. Int J Comput Assist Radiol Surg 2024; 19:69-82. [PMID: 37620748 PMCID: PMC10770268 DOI: 10.1007/s11548-023-03004-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/10/2023] [Accepted: 07/17/2023] [Indexed: 08/26/2023]
Abstract
PURPOSE For the modeling, execution, and control of complex, non-standardized intraoperative processes, a modeling language is needed that reflects the variability of interventions. As the established Business Process Model and Notation (BPMN) reaches its limits in terms of flexibility, the Case Management Model and Notation (CMMN) was considered as it addresses weakly structured processes. METHODS To analyze the suitability of the modeling languages, BPMN and CMMN models of a Robot-Assisted Minimally Invasive Esophagectomy and Cochlea Implantation were derived and integrated into a situation recognition workflow. Test cases were used to contrast the differences and compare the advantages and disadvantages of the models concerning modeling, execution, and control. Furthermore, the impact on transferability was investigated. RESULTS Compared to BPMN, CMMN allows flexibility for modeling intraoperative processes while remaining understandable. Although more effort and process knowledge are needed for execution and control within a situation recognition system, CMMN enables better transferability of the models and therefore the system. Concluding, CMMN should be chosen as a supplement to BPMN for flexible process parts that can only be covered insufficiently by BPMN, or otherwise as a replacement for the entire process. CONCLUSION CMMN offers the flexibility for variable, weakly structured process parts, and is thus suitable for surgical interventions. A combination of both notations could allow optimal use of their advantages and support the transferability of the situation recognition system.
Collapse
Affiliation(s)
- Denise Junger
- School of Informatics, Research Group Computer Assisted Medicine (CaMed), Reutlingen University, Reutlingen, Germany.
| | - Elisaveta Just
- School of Informatics, Research Group Computer Assisted Medicine (CaMed), Reutlingen University, Reutlingen, Germany
| | - Johanna M Brandenburg
- Department of General, Visceral and Transplantation Surgery, Heidelberg University Hospital, Heidelberg, Germany
- National Center for Tumor Diseases Heidelberg, Heidelberg, Germany
| | - Martin Wagner
- Department of General, Visceral and Transplantation Surgery, Heidelberg University Hospital, Heidelberg, Germany
- National Center for Tumor Diseases Heidelberg, Heidelberg, Germany
- Center for the Tactile Internet With Human in the Loop (CeTI), Technical University Dresden, Dresden, Germany
| | - Katharina Schaumann
- Department of Otorhinolaryngology, University Hospital Düsseldorf, Düsseldorf, Germany
| | - Thomas Klenzner
- Department of Otorhinolaryngology, University Hospital Düsseldorf, Düsseldorf, Germany
| | - Oliver Burgert
- School of Informatics, Research Group Computer Assisted Medicine (CaMed), Reutlingen University, Reutlingen, Germany
| |
Collapse
|
4
|
Zon M, Ganesh G, Deen MJ, Fang Q. Context-Aware Medical Systems within Healthcare Environments: A Systematic Scoping Review to Identify Subdomains and Significant Medical Contexts. INTERNATIONAL JOURNAL OF ENVIRONMENTAL RESEARCH AND PUBLIC HEALTH 2023; 20:6399. [PMID: 37510631 PMCID: PMC10379857 DOI: 10.3390/ijerph20146399] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/03/2023] [Revised: 06/24/2023] [Accepted: 07/04/2023] [Indexed: 07/30/2023]
Abstract
Context awareness is a field in pervasive computing, which has begun to impact medical systems via an increasing number of healthcare applications that are starting to use context awareness. The present work seeks to determine which contexts are important for medical applications and which domains of context-aware applications exist in healthcare. A systematic scoping review of context-aware medical systems currently used by patients or healthcare providers (inclusion criteria) was conducted between April 2021 and June 2023. A search strategy was designed and applied to Pub Med, EBSCO, IEEE Explore, Wiley, Science Direct, Springer Link, and ACM, articles from the databases were then filtered based on their abstract, and relevant articles were screened using a questionnaire applied to their full texts prior to data extraction. Applications were grouped into context-aware healthcare application domains based on past reviews and screening results. A total of 25 articles passed all screening levels and underwent data extraction. The most common contexts used were user location (8 out of 25 studies), demographic information (6 out of 25 studies), movement status/activity level (7 out of 25 studies), time of day (5 out of 25 studies), phone usage patterns (5 out of 25 studies), lab/vitals (7 out of 25 studies), and patient history data (8 out of 23 studies). Through a systematic review process, the current study determined the key contexts within context-aware healthcare applications that have reached healthcare providers and patients. The present work has illuminated many of the early successful context-aware healthcare applications. Additionally, the primary contexts leveraged by these systems have been identified, allowing future systems to focus on prioritizing the integration of these key contexts.
Collapse
Affiliation(s)
- Michael Zon
- Michael DeGroote School of Medicine, McMaster University, Hamilton, ON L8S 4L8, Canada
- School of Biomedical Engineering, McMaster University, Hamilton, ON L8S 4L8, Canada
| | - Guha Ganesh
- School of Biomedical Engineering, McMaster University, Hamilton, ON L8S 4L8, Canada
| | - M Jamal Deen
- Department of Electrical and Computer Engineering, McMaster University, Hamilton, ON L8S 4L8, Canada
| | - Qiyin Fang
- School of Biomedical Engineering, McMaster University, Hamilton, ON L8S 4L8, Canada
- Department of Engineering Physics, McMaster University, Hamilton, ON L8S 4L8, Canada
| |
Collapse
|
5
|
von Schudnat C, Schoeneberg KP, Albors-Garrigos J, Lahmann B, De-Miguel-Molina M. The Economic Impact of Standardization and Digitalization in the Operating Room: A Systematic Literature Review. J Med Syst 2023; 47:55. [PMID: 37129717 DOI: 10.1007/s10916-023-01945-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/31/2022] [Accepted: 03/29/2023] [Indexed: 05/03/2023]
Abstract
Hospital face increased resource constraints and competition. This escalates the need for efficiency optimization especially in resource-intense areas, such as the Operating Room (OR). Efficiency cannot happen at expenses of patient outcomes. Innovative digital support systems (DSS) have been introduced into the market to support established standardization methods of intraoperative workflows further. This review aimed to analyze whether applied standardization methods and implemented DSS of intraoperative surgical workflows lead to increasing efficiency and demonstrate economic improvements. A systematic review of intraoperative surgical workflows standardization and digitalization was performed. Journal articles and reviews from 2000 to 2023 were retrieved from EBSCO, PubMed, and Scopus databases, as well as the internal database of Johnson & Johnson. 17 articles showed a significant increase in efficiency through standardization, which led to cost reductions between $70.20 to $3,516 per case without negatively impacting quality. Five additional articles on DSS demonstrated a significant positive impact on efficiency and quality. Reduction in OR-time between 6 to 22% per case was one main contributor. No literature on DSS revealed any correlated economic impact. Selected standardization methods and introduced DSS for intraoperative surgical workflows effectively increase efficiency while maintaining or even improving quality. Demonstrated cost-effectiveness of non-digital standardization methods across surgical areas requires more research on complex and resource-intensive procedures and the economic value of DSS to support hospital management's strategic decisions to overcome the increasing economic burden.
Collapse
Affiliation(s)
- Christian von Schudnat
- Department of Business Organization, Faculty of Business Management, Universitat Politecnica de Valencia, Cami de Vera, s/n, 46022, Valencia, Spain.
| | - Klaus-Peter Schoeneberg
- Department of Economic and Social Sciences, Berliner Hochschule für Technik, Berlin, Luxemburger Str. 10, 13353, Berlin, Germany
| | - Jose Albors-Garrigos
- Department of Business Organization, Faculty of Business Management, Universitat Politecnica de Valencia, Cami de Vera, s/n, 46022, Valencia, Spain
| | - Benjamin Lahmann
- Department of Statistics and Operation Analysis, Faculty of Business and Economics, Mendel University Brno, Zemědělská 1, 61300, Brno, Czech Republic
| | - María De-Miguel-Molina
- Department of Business Organization, Faculty of Business Management, Universitat Politecnica de Valencia, Cami de Vera, s/n, 46022, Valencia, Spain
| |
Collapse
|
6
|
Ryniak C, Frommer SM, Junger D, Lohmann S, Stadelmaier M, Schmutz P, Stenzl A, Hirt B, Burgert O. A high-fidelity prototype of a sterile information system for the perioperative area: OR-Pad. Int J Comput Assist Radiol Surg 2023; 18:575-585. [PMID: 36370329 PMCID: PMC9939502 DOI: 10.1007/s11548-022-02787-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/10/2022] [Accepted: 10/25/2022] [Indexed: 11/13/2022]
Abstract
PURPOSE Supporting the surgeon during surgery is one of the main goals of intelligent ORs. The OR-Pad project aims to optimize the information flow within the perioperative area. A shared information space should enable appropriate preparation and provision of relevant information at any time before, during, and after surgery. METHODS Based on previous work on an interaction concept and system architecture for the sterile OR-Pad system, we designed a user interface for mobile and intraoperative (stationary) use, focusing on the most important functionalities like clear information provision to reduce information overload. The concepts were transferred into a high-fidelity prototype for demonstration purposes. The prototype was evaluated from different perspectives, including a usability study. RESULTS The prototype's central element is a timeline displaying all available case information chronologically, like radiological images, labor findings, or notes. This information space can be adapted for individual purposes (e.g., highlighting a tumor, filtering for own material). With the mobile and intraoperative mode of the system, relevant information can be added, preselected, viewed, and extended during the perioperative process. Overall, the evaluation showed good results and confirmed the vision of the information system. CONCLUSION The high-fidelity prototype of the information system OR-Pad focuses on supporting the surgeon via a timeline making all available case information accessible before, during, and after surgery. The information space can be personalized to enable targeted support. Further development is reasonable to optimize the approach and address missing or insufficient aspects, like the holding arm and sterility concept or new desired features.
Collapse
Affiliation(s)
- C Ryniak
- School of Informatics, Research Group Computer Assisted Medicine (CaMed), Reutlingen University, Reutlingen, Germany
| | - S M Frommer
- School of Informatics, Research Group Computer Assisted Medicine (CaMed), Reutlingen University, Reutlingen, Germany
| | - D Junger
- School of Informatics, Research Group Computer Assisted Medicine (CaMed), Reutlingen University, Reutlingen, Germany.
| | - S Lohmann
- School of Informatics, Research Group Computer Assisted Medicine (CaMed), Reutlingen University, Reutlingen, Germany
| | - M Stadelmaier
- School of Informatics, Research Group Computer Assisted Medicine (CaMed), Reutlingen University, Reutlingen, Germany
| | - P Schmutz
- School of Informatics, Research Group Computer Assisted Medicine (CaMed), Reutlingen University, Reutlingen, Germany
| | - A Stenzl
- Department of Urology, University Hospital Tübingen, Tübingen, Germany
| | - B Hirt
- Faculty of Medicine, Department of Anatomy, Institute for Clinical Anatomy and Cell Analytics, Eberhard Karls University Tübingen, Tübingen, Germany
| | - O Burgert
- School of Informatics, Research Group Computer Assisted Medicine (CaMed), Reutlingen University, Reutlingen, Germany
| |
Collapse
|
7
|
Berger J, Unger M, Keller J, Reich CM, Neumuth T, Melzer A. Design and validation of a medical robotic device system to control two collaborative robots for ultrasound-guided needle insertions. Front Robot AI 2022; 9:875845. [PMID: 36246494 PMCID: PMC9554707 DOI: 10.3389/frobt.2022.875845] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/14/2022] [Accepted: 08/31/2022] [Indexed: 11/13/2022] Open
Abstract
The percutaneous biopsy is a critical intervention for diagnosis and staging in cancer therapy. Robotic systems can improve the efficiency and outcome of such procedures while alleviating stress for physicians and patients. However, the high complexity of operation and the limited possibilities for robotic integration in the operating room (OR) decrease user acceptance and the number of deployed robots. Collaborative systems and standardized device communication may provide approaches to overcome named problems. Derived from the IEEE 11073 SDC standard terminology of medical device systems, we designed and validated a medical robotic device system (MERODES) to access and control a collaborative setup of two KUKA robots for ultrasound-guided needle insertions. The system is based on a novel standard for service-oriented device connectivity and utilizes collaborative principles to enhance user experience. Implementing separated workflow applications allows for a flexible system setup and configuration. The system was validated in three separate test scenarios to measure accuracies for 1) co-registration, 2) needle target planning in a water bath and 3) in an abdominal phantom. The co-registration accuracy averaged 0.94 ± 0.42 mm. The positioning errors ranged from 0.86 ± 0.42 to 1.19 ± 0.70 mm in the water bath setup and from 1.69 ± 0.92 to 1.96 ± 0.86 mm in the phantom. The presented results serve as a proof-of-concept and add to the current state of the art to alleviate system deployment and fast configuration for percutaneous robotic interventions.
Collapse
Affiliation(s)
- Johann Berger
- Innovation Center Computer Assisted Surgery (ICCAS), University of Leipzig, Leipzig, Germany
| | - Michael Unger
- Innovation Center Computer Assisted Surgery (ICCAS), University of Leipzig, Leipzig, Germany
| | - Johannes Keller
- Innovation Center Computer Assisted Surgery (ICCAS), University of Leipzig, Leipzig, Germany
| | - C. Martin Reich
- Innovation Center Computer Assisted Surgery (ICCAS), University of Leipzig, Leipzig, Germany
| | - Thomas Neumuth
- Innovation Center Computer Assisted Surgery (ICCAS), University of Leipzig, Leipzig, Germany
| | - Andreas Melzer
- Innovation Center Computer Assisted Surgery (ICCAS), University of Leipzig, Leipzig, Germany
- Institute for Medical Science and Technology (IMSaT), University Dundee, Dundee, Scotland
- *Correspondence: Andreas Melzer,
| |
Collapse
|
8
|
Junger D, Frommer SM, Burgert O. State-of-the-art of situation recognition systems for intraoperative procedures. Med Biol Eng Comput 2022; 60:921-939. [PMID: 35178622 PMCID: PMC8933302 DOI: 10.1007/s11517-022-02520-4] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/16/2020] [Accepted: 01/30/2022] [Indexed: 11/05/2022]
Abstract
One of the key challenges for automatic assistance is the support of actors in the operating room depending on the status of the procedure. Therefore, context information collected in the operating room is used to gain knowledge about the current situation. In literature, solutions already exist for specific use cases, but it is doubtful to what extent these approaches can be transferred to other conditions. We conducted a comprehensive literature research on existing situation recognition systems for the intraoperative area, covering 274 articles and 95 cross-references published between 2010 and 2019. We contrasted and compared 58 identified approaches based on defined aspects such as used sensor data or application area. In addition, we discussed applicability and transferability. Most of the papers focus on video data for recognizing situations within laparoscopic and cataract surgeries. Not all of the approaches can be used online for real-time recognition. Using different methods, good results with recognition accuracies above 90% could be achieved. Overall, transferability is less addressed. The applicability of approaches to other circumstances seems to be possible to a limited extent. Future research should place a stronger focus on adaptability. The literature review shows differences within existing approaches for situation recognition and outlines research trends. Applicability and transferability to other conditions are less addressed in current work.
Collapse
Affiliation(s)
- D Junger
- School of Informatics, Research Group Computer Assisted Medicine (CaMed), Reutlingen University, Alteburgstr. 150, 72762, Reutlingen, Germany.
| | - S M Frommer
- School of Informatics, Research Group Computer Assisted Medicine (CaMed), Reutlingen University, Alteburgstr. 150, 72762, Reutlingen, Germany
| | - O Burgert
- School of Informatics, Research Group Computer Assisted Medicine (CaMed), Reutlingen University, Alteburgstr. 150, 72762, Reutlingen, Germany
| |
Collapse
|
9
|
Yeh HH, Jain AM, Fox O, Wang SY. PhacoTrainer: A Multicenter Study of Deep Learning for Activity Recognition in Cataract Surgical Videos. Transl Vis Sci Technol 2021; 10:23. [PMID: 34784415 PMCID: PMC8606857 DOI: 10.1167/tvst.10.13.23] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/24/2022] Open
Abstract
Purpose To build and evaluate deep learning models for recognizing cataract surgical steps from whole-length surgical videos with minimal preprocessing, including identification of routine and complex steps. Methods We collected 298 cataract surgical videos from 12 resident surgeons across 6 sites and excluded 30 incomplete, duplicated, and combination surgery videos. Videos were downsampled at 1 frame/second. Trained annotators labeled 13 steps of surgery: create wound, injection into the eye, capsulorrhexis, hydrodissection, phacoemulsification, irrigation/aspiration, place lens, remove viscoelastic, close wound, advanced technique/other, stain with trypan blue, manipulating iris, and subconjunctival injection. We trained two deep learning models, one based on the VGG16 architecture (VGG model) and the second using VGG16 followed by a long short-term memory network (convolutional neural network [CNN]– recurrent neural network [RNN] model). Class activation maps were visualized using Grad-CAM. Results Overall top 1 prediction accuracy was 76% for VGG model (93% for top 3 accuracy) and 84% for the CNN–RNN model (97% for top 3 accuracy). The microaveraged area under receiver-operating characteristic curves was 0.97 for the VGG model and 0.99 for the CNN–RNN model. The microaveraged average precision score was 0.83 for the VGG model and 0.92 for the CNN–RNN model. Class activation maps revealed the model was appropriately focused on the instrumentation used in each step to identify which step was being performed. Conclusions Deep learning models can classify cataract surgical activities on a frame-by-frame basis with remarkably high accuracy, especially routine surgical steps. Translational Relevance An automated system for recognition of cataract surgical steps could provide to residents automated feedback metrics, such as the length of time spent on each step.
Collapse
Affiliation(s)
- Hsu-Hang Yeh
- Department of Biomedical Data Science, Stanford University, Palo Alto, CA, USA
| | - Anjal M Jain
- Department of Ophthalmology, Byers Eye Institute, Stanford University, Palo Alto, CA, USA
| | - Olivia Fox
- Krieger School of Arts and Sciences, Johns Hopkins University, Baltimore, MD, USA
| | - Sophia Y Wang
- Department of Biomedical Data Science, Stanford University, Palo Alto, CA, USA.,Department of Ophthalmology, Byers Eye Institute, Stanford University, Palo Alto, CA, USA
| |
Collapse
|
10
|
Weber-Chüo T, Rockstroh M, Franke S, Hofer M, Dietz A, Neumuth T, Pirlich M. [Evaluation of an integrated OR based on open standards in Cochlea Implant surgery]. Laryngorhinootologie 2021; 100:987-996. [PMID: 33494113 DOI: 10.1055/a-1346-9227] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/22/2022]
Abstract
BACKGROUND Digitalization in surgery makes it necessary to develop modern surgical concepts. New approaches to system networking with integration and open standardized communication of all medical devices are being pursued. METHODS At the University Hospital Leipzig, a demonstration of the integrated OR was carried out together with the Innovation Center for Computer Assisted Surgery (ICCAS) using the example of a cochlea implantation. The preoperative management, technical preparation, surgical procedure and postoperative documentation by a total of n = 30 study participants (2 expert groups) were evaluated. In addition to the collection of objective parameters, qualitative questionnaires and quantitative, interval-scaled questions were used. RESULTS Preoperatively, the digital presentation of the patient's clinical data was considered as helpful by both groups (group 1: median = 5, group 2: median = 4). This also applies to the personalized OR settings, the intraoperative display options and the dynamic, surgeon-centered visualization (median = 4). Similar positive conclusions were drawn for postoperative documentation and postoperative follow-up (median = 4). A significant difference in the final evaluation of the integrated surgical concept between the two expert groups could not be determined (p > 0.05). CONCLUSIONS The positive study results show that the theoretical idea of system networking based on open standards can be successfully implemented in practice using the example of a cochlea implantation. Thus, the intelligent "operating room of the future" no longer seems to be a fictitious idea, but a realistic image of modern surgical medicine.
Collapse
Affiliation(s)
- Teresa Weber-Chüo
- Klinik und Poliklinik für Hals-Nasen-Ohrenheilkunde, Plastische Operationen, Universitätsklinikum Leipzig, Germany
| | - Max Rockstroh
- Innovation Center Computer Assisted Surgery (ICCAS), Universität Leipzig, Germany
| | - Stefan Franke
- Innovation Center Computer Assisted Surgery (ICCAS), Universität Leipzig, Germany
| | | | - Andreas Dietz
- Klinik und Poliklinik für Hals-Nasen-Ohrenheilkunde, Plastische Operationen, Universitätsklinikum Leipzig, Germany
| | - Thomas Neumuth
- Innovation Center Computer Assisted Surgery (ICCAS), Universität Leipzig, Germany
| | - Markus Pirlich
- Klinik und Poliklinik für Hals-Nasen-Ohrenheilkunde, Plastische Operationen, Universitätsklinikum Leipzig, Germany
| |
Collapse
|
11
|
Kitaguchi D, Takeshita N, Matsuzaki H, Oda T, Watanabe M, Mori K, Kobayashi E, Ito M. Automated laparoscopic colorectal surgery workflow recognition using artificial intelligence: Experimental research. Int J Surg 2020; 79:88-94. [PMID: 32413503 DOI: 10.1016/j.ijsu.2020.05.015] [Citation(s) in RCA: 26] [Impact Index Per Article: 6.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/06/2020] [Revised: 05/01/2020] [Accepted: 05/05/2020] [Indexed: 12/13/2022]
Abstract
BACKGROUND Identifying laparoscopic surgical videos using artificial intelligence (AI) facilitates the automation of several currently time-consuming manual processes, including video analysis, indexing, and video-based skill assessment. This study aimed to construct a large annotated dataset comprising laparoscopic colorectal surgery (LCRS) videos from multiple institutions and evaluate the accuracy of automatic recognition for surgical phase, action, and tool by combining this dataset with AI. MATERIALS AND METHODS A total of 300 intraoperative videos were collected from 19 high-volume centers. A series of surgical workflows were classified into 9 phases and 3 actions, and the area of 5 tools were assigned by painting. More than 82 million frames were annotated for a phase and action classification task, and 4000 frames were annotated for a tool segmentation task. Of these frames, 80% were used for the training dataset and 20% for the test dataset. A convolutional neural network (CNN) was used to analyze the videos. Intersection over union (IoU) was used as the evaluation metric for tool recognition. RESULTS The overall accuracies for the automatic surgical phase and action classification task were 81.0% and 83.2%, respectively. The mean IoU for the automatic tool segmentation task for 5 tools was 51.2%. CONCLUSIONS A large annotated dataset of LCRS videos was constructed, and the phase, action, and tool were recognized with high accuracy using AI. Our dataset has potential uses in medical applications such as automatic video indexing and surgical skill assessments. Open research will assist in improving CNN models by making our dataset available in the field of computer vision.
Collapse
Affiliation(s)
- Daichi Kitaguchi
- Surgical Device Innovation Office, National Cancer Center Hospital East, 6-5-1 Kashiwanoha, Kashiwa, Chiba, 277-8577, Japan; Department of Colorectal Surgery, National Cancer Center Hospital East, 6-5-1 Kashiwanoha, Kashiwa, Chiba, 277-8577, Japan; Department of Gastrointestinal and Hepato-Biliary-Pancreatic Surgery, Faculty of Medicine, University of Tsukuba, 1-1-1 Tennodai, Tsukuba, Ibaraki, 305-8575, Japan
| | - Nobuyoshi Takeshita
- Surgical Device Innovation Office, National Cancer Center Hospital East, 6-5-1 Kashiwanoha, Kashiwa, Chiba, 277-8577, Japan; Department of Colorectal Surgery, National Cancer Center Hospital East, 6-5-1 Kashiwanoha, Kashiwa, Chiba, 277-8577, Japan.
| | - Hiroki Matsuzaki
- Surgical Device Innovation Office, National Cancer Center Hospital East, 6-5-1 Kashiwanoha, Kashiwa, Chiba, 277-8577, Japan
| | - Tatsuya Oda
- Department of Gastrointestinal and Hepato-Biliary-Pancreatic Surgery, Faculty of Medicine, University of Tsukuba, 1-1-1 Tennodai, Tsukuba, Ibaraki, 305-8575, Japan
| | - Masahiko Watanabe
- Department of Surgery, Kitasato University School of Medicine, 1-15-1 Kitasato, Minami-ku, Sagamihara, Kanagawa, 252-0374, Japan
| | - Kensaku Mori
- Graduate School of Informatics, Nagoya University, Furo-cho, Chikusa-ku, Nagoya, Aichi, 464-8601, Japan
| | - Etsuko Kobayashi
- Institute of Advanced Biomedical Engineering and Science, Tokyo Women's Medical University, 8-1 Kawada-cho, Shinjuku-ku, Tokyo, 162-8666, Japan
| | - Masaaki Ito
- Surgical Device Innovation Office, National Cancer Center Hospital East, 6-5-1 Kashiwanoha, Kashiwa, Chiba, 277-8577, Japan; Department of Colorectal Surgery, National Cancer Center Hospital East, 6-5-1 Kashiwanoha, Kashiwa, Chiba, 277-8577, Japan.
| |
Collapse
|
12
|
Neumann J, Franke S, Rockstroh M, Kasparick M, Neumuth T. Extending BPMN 2.0 for intraoperative workflow modeling with IEEE 11073 SDC for description and orchestration of interoperable, networked medical devices. Int J Comput Assist Radiol Surg 2019; 14:1403-1413. [PMID: 31055764 DOI: 10.1007/s11548-019-01982-6] [Citation(s) in RCA: 11] [Impact Index Per Article: 2.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/10/2019] [Accepted: 04/16/2019] [Indexed: 11/29/2022]
Abstract
PURPOSE Surgical workflow management in integrated operating rooms (ORs) enables the implementation of novel computer-aided surgical assistance and new applications in process automation, situation awareness, and decision support. The context-sensitive configuration and orchestration of interoperable, networked medical devices is a prerequisite for an effective reduction in the surgeons' workload, by providing the right service and right information at the right time. The information about the surgical situation must be described as surgical process models and distributed to the medical devices and IT systems in the OR. Available modeling languages are not capable of describing surgical processes for this application. METHODS In this work, the BPMNSIX modeling language for intraoperative processes is technically enhanced and implemented for workflow build-time and run-time. Therefore, particular attention is given to the integration of the recently published IEEE 11073 SDC standard family for a service-oriented architecture of networked medical devices. In addition, interaction patterns for context-aware configuration and device orchestration were presented. RESULTS The identified interaction patterns were implemented in BPMNSIX for an ophthalmologic use case. Therefore, the examples of the process-driven incorporation and control of device services could be demonstrated. CONCLUSION The modeling of surgical procedures with BPMNSIX allows the implementation of context-sensitive surgical assistance functionalities and enables flexibility in terms of the orchestration of dynamically changing device ensembles and integration of unknown devices in the surgical workflow management.
Collapse
Affiliation(s)
- Juliane Neumann
- Innovation Center Computer Assisted Surgery (ICCAS), Leipzig University, Semmelweisstr. 14, 04103, Leipzig, Germany.
| | - Stefan Franke
- Innovation Center Computer Assisted Surgery (ICCAS), Leipzig University, Semmelweisstr. 14, 04103, Leipzig, Germany
| | - Max Rockstroh
- Innovation Center Computer Assisted Surgery (ICCAS), Leipzig University, Semmelweisstr. 14, 04103, Leipzig, Germany
| | - Martin Kasparick
- Institute of Applied Microelectronics and Computer Engineering, University of Rostock, Rostock, Germany
| | - Thomas Neumuth
- Innovation Center Computer Assisted Surgery (ICCAS), Leipzig University, Semmelweisstr. 14, 04103, Leipzig, Germany
| |
Collapse
|
13
|
Kasparick M, Andersen B, Franke S, Rockstroh M, Golatowski F, Timmermann D, Ingenerf J, Neumuth T. Enabling artificial intelligence in high acuity medical environments. MINIM INVASIV THER 2019; 28:120-126. [PMID: 30950665 DOI: 10.1080/13645706.2019.1599957] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/27/2022]
Abstract
Acute patient treatment can heavily profit from AI-based assistive and decision support systems, in terms of improved patient outcome as well as increased efficiency. Yet, only very few applications have been reported because of the limited accessibility of device data due to the lack of adoption of open standards, and the complexity of regulatory/approval requirements for AI-based systems. The fragmentation of data, still being stored in isolated silos, results in limited accessibility for AI in healthcare and machine learning is complicated by the loss of semantics in data conversions. We outline a reference model that addresses the requirements of innovative AI-based research systems as well as the clinical reality. The integration of networked medical devices and Clinical Repositories based on open standards, such as IEEE 11073 SDC and HL7 FHIR, will foster novel assistance and decision support. The reference model will make point-of-care device data available for AI-based approaches. Semantic interoperability between Clinical and Research Repositories will allow correlating patient data, device data, and the patient outcome. Thus, complete workflows in high acuity environments can be analysed. Open semantic interoperability will enable the improvement of patient outcome and the increase of efficiency on a large scale and across clinical applications.
Collapse
Affiliation(s)
- Martin Kasparick
- a Institute of Applied Microelectronics and Computer Engineering (IMD) , University of Rostock , Rostock , Germany
| | - Björn Andersen
- b Institute of Medical Informatics , University of Lübeck , Lübeck , Germany
| | - Stefan Franke
- c Innovation Center Computer Assisted Surgery (ICCAS) , University of Leipzig , Leipzig , Germany
| | - Max Rockstroh
- c Innovation Center Computer Assisted Surgery (ICCAS) , University of Leipzig , Leipzig , Germany
| | - Frank Golatowski
- a Institute of Applied Microelectronics and Computer Engineering (IMD) , University of Rostock , Rostock , Germany
| | - Dirk Timmermann
- a Institute of Applied Microelectronics and Computer Engineering (IMD) , University of Rostock , Rostock , Germany
| | - Josef Ingenerf
- b Institute of Medical Informatics , University of Lübeck , Lübeck , Germany
| | - Thomas Neumuth
- c Innovation Center Computer Assisted Surgery (ICCAS) , University of Leipzig , Leipzig , Germany
| |
Collapse
|