1
|
Huaulmé A, Harada K, Nguyen QM, Park B, Hong S, Choi MK, Peven M, Li Y, Long Y, Dou Q, Kumar S, Lalithkumar S, Hongliang R, Matsuzaki H, Ishikawa Y, Harai Y, Kondo S, Mitsuishi M, Jannin P. PEg TRAnsfer Workflow recognition challenge report: Do multimodal data improve recognition? Comput Methods Programs Biomed 2023; 236:107561. [PMID: 37119774 DOI: 10.1016/j.cmpb.2023.107561] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/19/2022] [Revised: 04/06/2023] [Accepted: 04/18/2023] [Indexed: 05/21/2023]
Abstract
BACKGROUND AND OBJECTIVE In order to be context-aware, computer-assisted surgical systems require accurate, real-time automatic surgical workflow recognition. In the past several years, surgical video has been the most commonly-used modality for surgical workflow recognition. But with the democratization of robot-assisted surgery, new modalities, such as kinematics, are now accessible. Some previous methods use these new modalities as input for their models, but their added value has rarely been studied. This paper presents the design and results of the "PEg TRAnsfer Workflow recognition" (PETRAW) challenge with the objective of developing surgical workflow recognition methods based on one or more modalities and studying their added value. METHODS The PETRAW challenge included a data set of 150 peg transfer sequences performed on a virtual simulator. This data set included videos, kinematic data, semantic segmentation data, and annotations, which described the workflow at three levels of granularity: phase, step, and activity. Five tasks were proposed to the participants: three were related to the recognition at all granularities simultaneously using a single modality, and two addressed the recognition using multiple modalities. The mean application-dependent balanced accuracy (AD-Accuracy) was used as an evaluation metric to take into account class balance and is more clinically relevant than a frame-by-frame score. RESULTS Seven teams participated in at least one task with four participating in every task. The best results were obtained by combining video and kinematic data (AD-Accuracy of between 93% and 90% for the four teams that participated in all tasks). CONCLUSION The improvement of surgical workflow recognition methods using multiple modalities compared with unimodal methods was significant for all teams. However, the longer execution time required for video/kinematic-based methods(compared to only kinematic-based methods) must be considered. Indeed, one must ask if it is wise to increase computing time by 2000 to 20,000% only to increase accuracy by 3%. The PETRAW data set is publicly available at www.synapse.org/PETRAW to encourage further research in surgical workflow recognition.
Collapse
Affiliation(s)
- Arnaud Huaulmé
- Univ Rennes, INSERM, LTSI - UMR 1099, Rennes, F35000, France.
| | - Kanako Harada
- Department of Mechanical Engineering, the University of Tokyo, Tokyo 113-8656, Japan
| | | | - Bogyu Park
- VisionAI hutom, Seoul, Republic of Korea
| | | | | | | | | | - Yonghao Long
- Department of Computer Science & Engineering, The Chinese University of Hong Kong, Hong Kong
| | - Qi Dou
- Department of Computer Science & Engineering, The Chinese University of Hong Kong, Hong Kong
| | | | | | - Ren Hongliang
- National University of Singapore, Singapore, Singapore; The Chinese University of Hong Kong, Hong Kong, Hong Kong
| | - Hiroki Matsuzaki
- National Cancer Center Japan East Hospital, Tokyo 104-0045, Japan
| | - Yuto Ishikawa
- National Cancer Center Japan East Hospital, Tokyo 104-0045, Japan
| | - Yuriko Harai
- National Cancer Center Japan East Hospital, Tokyo 104-0045, Japan
| | | | - Manoru Mitsuishi
- Department of Mechanical Engineering, the University of Tokyo, Tokyo 113-8656, Japan
| | - Pierre Jannin
- Univ Rennes, INSERM, LTSI - UMR 1099, Rennes, F35000, France.
| |
Collapse
|
2
|
Perumalla C, Kearse L, Peven M, Laufer S, Goll C, Wise B, Yang S, Pugh C. AI-Based Video Segmentation: Procedural Steps or Basic Maneuvers? J Surg Res 2023; 283:500-506. [PMID: 36436286 PMCID: PMC10368211 DOI: 10.1016/j.jss.2022.10.069] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/29/2022] [Revised: 10/14/2022] [Accepted: 10/19/2022] [Indexed: 11/25/2022]
Abstract
INTRODUCTION Video-based review of surgical procedures has proven to be useful in training by enabling efficiency in the qualitative assessment of surgical skill and intraoperative decision-making. Current video segmentation protocols focus largely on procedural steps. Although some operations are more complex than others, many of the steps in any given procedure involve an intricate choreography of basic maneuvers such as suturing, knot tying, and cutting. The use of these maneuvers at certain procedural steps can convey information that aids in the assessment of the complexity of the procedure, surgical preference, and skill. Our study aims to develop and evaluate an algorithm to identify these maneuvers. METHODS A standard deep learning architecture was used to differentiate between suture throws, knot ties, and suture cutting on a data set comprised of videos from practicing clinicians (N = 52) who participated in a simulated enterotomy repair. Perception of the added value to traditional artificial intelligence segmentation was explored by qualitatively examining the utility of identifying maneuvers in a subset of steps for an open colon resection. RESULTS An accuracy of 84% was reached in differentiating maneuvers. The precision in detecting the basic maneuvers was 87.9%, 60%, and 90.9% for suture throws, knot ties, and suture cutting, respectively. The qualitative concept mapping confirmed realistic scenarios that could benefit from basic maneuver identification. CONCLUSIONS Basic maneuvers can indicate error management activity or safety measures and allow for the assessment of skill. Our deep learning algorithm identified basic maneuvers with reasonable accuracy. Such models can aid in artificial intelligence-assisted video review by providing additional information that can complement traditional video segmentation protocols.
Collapse
Affiliation(s)
- Calvin Perumalla
- Stanford School of Medicine, Department of Surgery, Stanford, California.
| | - LaDonna Kearse
- Stanford School of Medicine, Department of Surgery, Stanford, California
| | - Michael Peven
- John Hopkins University, Department of Computer Science, Baltimore, Maryland
| | - Shlomi Laufer
- Industrial Engineering and Management, Technion - Israel Institute of Technology, Haifa, Israel
| | - Cassidi Goll
- Stanford School of Medicine, Department of Surgery, Stanford, California
| | - Brett Wise
- Stanford School of Medicine, Department of Surgery, Stanford, California
| | - Su Yang
- Stanford School of Medicine, Department of Surgery, Stanford, California
| | - Carla Pugh
- Stanford School of Medicine, Department of Surgery, Stanford, California
| |
Collapse
|
3
|
Rawat N, Rao V, Peven M, Shrock C, Reiter A, Saria S, Ali H. Comparison of Automated Activity Recognition to Provider Observations of Patient Mobility in the ICU. Crit Care Med 2020; 47:1232-1234. [PMID: 31162207 DOI: 10.1097/ccm.0000000000003852] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/25/2022]
Abstract
OBJECTIVES To compare noninvasive mobility sensor patient motion signature to direct observations by physicians and nurses. DESIGN Prospective, observational study. SETTING Academic hospital surgical ICU. PATIENTS AND MEASUREMENTS A total of 2,426 1-minute clips from six ICU patients (development dataset) and 4,824 1-minute clips from five patients (test dataset). INTERVENTIONS None. MAIN RESULTS Noninvasive mobility sensor achieved a minute-level accuracy of 94.2% (2,138/2,272) and an hour-level accuracy of 81.4% (70/86). CONCLUSIONS The automated noninvasive mobility sensor system represents a significant departure from current manual measurement and reporting used in clinical care, lowering the burden of measurement and documentation on caregivers.
Collapse
Affiliation(s)
- Nishi Rawat
- Department of Anesthesia and Critical Care Medicine, Johns Hopkins University School of Medicine, Baltimore, MD.,Armstrong Institute for Patient Safety and Quality, Johns Hopkins University School of Medicine, Baltimore, MD
| | - Vishal Rao
- Department of Computer Science, Johns Hopkins University, Baltimore, MD
| | - Michael Peven
- Department of Computer Science, Johns Hopkins University, Baltimore, MD
| | | | - Austin Reiter
- Department of Computer Science, Johns Hopkins University, Baltimore, MD
| | - Suchi Saria
- Armstrong Institute for Patient Safety and Quality, Johns Hopkins University School of Medicine, Baltimore, MD.,Department of Computer Science, Johns Hopkins University, Baltimore, MD.,Department of Health Policy and Management, Johns Hopkins University Bloomberg School of Public Health, Baltimore, MD
| | - Haider Ali
- Department of Computer Science, Johns Hopkins University, Baltimore, MD
| |
Collapse
|