1
|
Yoshida M, Kitaguchi D, Takeshita N, Matsuzaki H, Ishikawa Y, Yura M, Akimoto T, Kinoshita T, Ito M. Surgical step recognition in laparoscopic distal gastrectomy using artificial intelligence: a proof-of-concept study. Langenbecks Arch Surg 2024; 409:213. [PMID: 38995411 DOI: 10.1007/s00423-024-03411-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/27/2024] [Accepted: 07/05/2024] [Indexed: 07/13/2024]
Abstract
PURPOSE Laparoscopic distal gastrectomy (LDG) is a difficult procedure for early career surgeons. Artificial intelligence (AI)-based surgical step recognition is crucial for establishing context-aware computer-aided surgery systems. In this study, we aimed to develop an automatic recognition model for LDG using AI and evaluate its performance. METHODS Patients who underwent LDG at our institution in 2019 were included in this study. Surgical video data were classified into the following nine steps: (1) Port insertion; (2) Lymphadenectomy on the left side of the greater curvature; (3) Lymphadenectomy on the right side of the greater curvature; (4) Division of the duodenum; (5) Lymphadenectomy of the suprapancreatic area; (6) Lymphadenectomy on the lesser curvature; (7) Division of the stomach; (8) Reconstruction; and (9) From reconstruction to completion of surgery. Two gastric surgeons manually assigned all annotation labels. Convolutional neural network (CNN)-based image classification was further employed to identify surgical steps. RESULTS The dataset comprised 40 LDG videos. Over 1,000,000 frames with annotated labels of the LDG steps were used to train the deep-learning model, with 30 and 10 surgical videos for training and validation, respectively. The classification accuracies of the developed models were precision, 0.88; recall, 0.87; F1 score, 0.88; and overall accuracy, 0.89. The inference speed of the proposed model was 32 ps. CONCLUSION The developed CNN model automatically recognized the LDG surgical process with relatively high accuracy. Adding more data to this model could provide a fundamental technology that could be used in the development of future surgical instruments.
Collapse
Affiliation(s)
- Mitsumasa Yoshida
- Gastric Surgery Division, National Cancer Center Hospital East, 6-5-1 Kashiwanoha, Kashiwa, Chiba, 277-8577, Japan
- Department for the Promotion of Medical Device Innovation, National Cancer Center Hospital East, 6- 5-1 Kashiwanoha, Kashiwa, Chiba, 277-8577, Japan
- Course of Advanced Clinical Research of Cancer, Juntendo University Graduate School of Medicine, 2- 1-1, Hongo, Bunkyo-Ward, Tokyo, 113-8421, Japan
| | - Daichi Kitaguchi
- Department for the Promotion of Medical Device Innovation, National Cancer Center Hospital East, 6- 5-1 Kashiwanoha, Kashiwa, Chiba, 277-8577, Japan
| | - Nobuyoshi Takeshita
- Department for the Promotion of Medical Device Innovation, National Cancer Center Hospital East, 6- 5-1 Kashiwanoha, Kashiwa, Chiba, 277-8577, Japan
| | - Hiroki Matsuzaki
- Department for the Promotion of Medical Device Innovation, National Cancer Center Hospital East, 6- 5-1 Kashiwanoha, Kashiwa, Chiba, 277-8577, Japan
| | - Yuto Ishikawa
- Department for the Promotion of Medical Device Innovation, National Cancer Center Hospital East, 6- 5-1 Kashiwanoha, Kashiwa, Chiba, 277-8577, Japan
| | - Masahiro Yura
- Gastric Surgery Division, National Cancer Center Hospital East, 6-5-1 Kashiwanoha, Kashiwa, Chiba, 277-8577, Japan
| | - Tetsuo Akimoto
- Course of Advanced Clinical Research of Cancer, Juntendo University Graduate School of Medicine, 2- 1-1, Hongo, Bunkyo-Ward, Tokyo, 113-8421, Japan
| | - Takahiro Kinoshita
- Gastric Surgery Division, National Cancer Center Hospital East, 6-5-1 Kashiwanoha, Kashiwa, Chiba, 277-8577, Japan
| | - Masaaki Ito
- Department for the Promotion of Medical Device Innovation, National Cancer Center Hospital East, 6- 5-1 Kashiwanoha, Kashiwa, Chiba, 277-8577, Japan.
| |
Collapse
|
2
|
Horita K, Hida K, Itatani Y, Fujita H, Hidaka Y, Yamamoto G, Ito M, Obama K. Real-time detection of active bleeding in laparoscopic colectomy using artificial intelligence. Surg Endosc 2024; 38:3461-3469. [PMID: 38760565 DOI: 10.1007/s00464-024-10874-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/12/2024] [Accepted: 04/20/2024] [Indexed: 05/19/2024]
Abstract
BACKGROUND Most intraoperative adverse events (iAEs) result from surgeons' errors, and bleeding is the majority of iAEs. Recognizing active bleeding timely is important to ensure safe surgery, and artificial intelligence (AI) has great potential for detecting active bleeding and providing real-time surgical support. This study aimed to develop a real-time AI model to detect active intraoperative bleeding. METHODS We extracted 27 surgical videos from a nationwide multi-institutional surgical video database in Japan and divided them at the patient level into three sets: training (n = 21), validation (n = 3), and testing (n = 3). We subsequently extracted the bleeding scenes and labeled distinctively active bleeding and blood pooling frame by frame. We used pre-trained YOLOv7_6w and developed a model to learn both active bleeding and blood pooling. The Average Precision at an Intersection over Union threshold of 0.5 (AP.50) for active bleeding and frames per second (FPS) were quantified. In addition, we conducted two 5-point Likert scales (5 = Excellent, 4 = Good, 3 = Fair, 2 = Poor, and 1 = Fail) questionnaires about sensitivity (the sensitivity score) and number of overdetection areas (the overdetection score) to investigate the surgeons' assessment. RESULTS We annotated 34,117 images of 254 bleeding events. The AP.50 for active bleeding in the developed model was 0.574 and the FPS was 48.5. Twenty surgeons answered two questionnaires, indicating a sensitivity score of 4.92 and an overdetection score of 4.62 for the model. CONCLUSIONS We developed an AI model to detect active bleeding, achieving real-time processing speed. Our AI model can be used to provide real-time surgical support.
Collapse
Affiliation(s)
- Kenta Horita
- Department of Surgery, Kyoto University Graduate School of Medicine, 54 Shogoin-Kawahara-Cho, Sakyo-Ku, Kyoto, 606-8507, Japan
| | - Koya Hida
- Department of Surgery, Kyoto University Graduate School of Medicine, 54 Shogoin-Kawahara-Cho, Sakyo-Ku, Kyoto, 606-8507, Japan.
| | - Yoshiro Itatani
- Department of Surgery, Kyoto University Graduate School of Medicine, 54 Shogoin-Kawahara-Cho, Sakyo-Ku, Kyoto, 606-8507, Japan
| | - Haruku Fujita
- Department of Surgery, Kyoto University Graduate School of Medicine, 54 Shogoin-Kawahara-Cho, Sakyo-Ku, Kyoto, 606-8507, Japan
| | - Yu Hidaka
- Department of Biomedical Statistics and Bioinformatics, Kyoto University Graduate School of Medicine, Kyoto, Japan
| | - Goshiro Yamamoto
- Division of Medical Information Technology and Administration Planning, Kyoto University, Kyoto, Japan
| | - Masaaki Ito
- Surgical Device Innovation Office, National Cancer Center Hospital East, Chiba, Japan
- Department of Colorectal Surgery, National Cancer Center Hospital East, Chiba, Japan
| | - Kazutaka Obama
- Department of Surgery, Kyoto University Graduate School of Medicine, 54 Shogoin-Kawahara-Cho, Sakyo-Ku, Kyoto, 606-8507, Japan
| |
Collapse
|
3
|
Hu L, Feng S, Wang B. Weakly Supervised Pose Estimation of Surgical Instrument from a Single Endoscopic Image. SENSORS (BASEL, SWITZERLAND) 2024; 24:3355. [PMID: 38894146 PMCID: PMC11174500 DOI: 10.3390/s24113355] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/28/2024] [Revised: 05/15/2024] [Accepted: 05/21/2024] [Indexed: 06/21/2024]
Abstract
Instrument pose estimation is a key demand in computer-aided surgery, and its main challenges lie in two aspects: Firstly, the difficulty of obtaining stable corresponding image feature points due to the instruments' high refraction and complicated background, and secondly, the lack of labeled pose data. This study aims to tackle the pose estimation problem of surgical instruments in the current endoscope system using a single endoscopic image. More specifically, a weakly supervised method based on the instrument's image segmentation contour is proposed, with the effective assistance of synthesized endoscopic images. Our method consists of the following three modules: a segmentation module to automatically detect the instrument in the input image, followed by a point inference module to predict the image locations of the implicit feature points of the instrument, and a point back-propagatable Perspective-n-Point module to estimate the pose from the tentative 2D-3D corresponding points. To alleviate the over-reliance on point correspondence accuracy, the local errors of feature point matching and the global inconsistency of the corresponding contours are simultaneously minimized. Our proposed method is validated with both real and synthetic images in comparison with the current state-of-the-art methods.
Collapse
Affiliation(s)
- Lihua Hu
- College of Computer Sciences and Technology, Taiyuan University of Science and Technology, Taiyuan 030024, China; (L.H.); (S.F.)
| | - Shida Feng
- College of Computer Sciences and Technology, Taiyuan University of Science and Technology, Taiyuan 030024, China; (L.H.); (S.F.)
- State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China
| | - Bo Wang
- State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China
| |
Collapse
|
4
|
Al Abbas AI, Namazi B, Radi I, Alterio R, Abreu AA, Rail B, Polanco PM, Zeh HJ, Hogg ME, Zureikat AH, Sankaranarayanan G. The development of a deep learning model for automated segmentation of the robotic pancreaticojejunostomy. Surg Endosc 2024; 38:2553-2561. [PMID: 38488870 DOI: 10.1007/s00464-024-10725-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/01/2023] [Accepted: 01/28/2024] [Indexed: 03/17/2024]
Abstract
BACKGROUND Minimally invasive surgery provides an unprecedented opportunity to review video for assessing surgical performance. Surgical video analysis is time-consuming and expensive. Deep learning provides an alternative for analysis. Robotic pancreaticoduodenectomy (RPD) is a complex and morbid operation. Surgeon technical performance of pancreaticojejunostomy (PJ) has been associated with postoperative pancreatic fistula. In this work, we aimed to utilize deep learning to automatically segment PJ RPD videos. METHODS This was a retrospective review of prospectively collected videos from 2011 to 2022 that were in libraries at tertiary referral centers, including 111 PJ videos. Each frame of a robotic PJ video was categorized based on 6 tasks. A 3D convolutional neural network was trained for frame-level visual feature extraction and classification. All the videos were manually annotated for the start and end of each task. RESULTS Of the 100 videos assessed, 60 videos were used for the training the model, 10 for hyperparameter optimization, and 30 for the testing of performance. All the frames were extracted (6 frames/second) and annotated. The accuracy and mean per-class F1 scores were 88.01% and 85.34% for tasks. CONCLUSION The deep learning model performed well for automated segmentation of PJ videos. Future work will focus on skills assessment and outcome prediction.
Collapse
Affiliation(s)
- Amr I Al Abbas
- Department of Surgery, University of Texas Southwestern Medical Center, 5323 Harry Hines Blvd., Dallas, TX, 75390-9169, USA
| | - Babak Namazi
- Department of Surgery, University of Texas Southwestern Medical Center, 5323 Harry Hines Blvd., Dallas, TX, 75390-9169, USA
| | - Imad Radi
- Department of Surgery, University of Texas Southwestern Medical Center, 5323 Harry Hines Blvd., Dallas, TX, 75390-9169, USA
| | - Rodrigo Alterio
- Department of Surgery, University of Texas Southwestern Medical Center, 5323 Harry Hines Blvd., Dallas, TX, 75390-9169, USA
| | - Andres A Abreu
- Department of Surgery, University of Texas Southwestern Medical Center, 5323 Harry Hines Blvd., Dallas, TX, 75390-9169, USA
| | - Benjamin Rail
- Department of Surgery, University of Texas Southwestern Medical Center, 5323 Harry Hines Blvd., Dallas, TX, 75390-9169, USA
| | - Patricio M Polanco
- Department of Surgery, University of Texas Southwestern Medical Center, 5323 Harry Hines Blvd., Dallas, TX, 75390-9169, USA
| | - Herbert J Zeh
- Department of Surgery, University of Texas Southwestern Medical Center, 5323 Harry Hines Blvd., Dallas, TX, 75390-9169, USA
| | | | - Amer H Zureikat
- University of Pittsburgh Medical Center, Pittsburgh, PA, USA
| | - Ganesh Sankaranarayanan
- Department of Surgery, University of Texas Southwestern Medical Center, 5323 Harry Hines Blvd., Dallas, TX, 75390-9169, USA.
| |
Collapse
|
5
|
Zhu Y, Du L, Fu PY, Geng ZH, Zhang DF, Chen WF, Li QL, Zhou PH. An Automated Video Analysis System for Retrospective Assessment and Real-Time Monitoring of Endoscopic Procedures (with Video). Bioengineering (Basel) 2024; 11:445. [PMID: 38790312 PMCID: PMC11118061 DOI: 10.3390/bioengineering11050445] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/05/2024] [Revised: 04/21/2024] [Accepted: 04/22/2024] [Indexed: 05/26/2024] Open
Abstract
BACKGROUND AND AIMS Accurate recognition of endoscopic instruments facilitates quantitative evaluation and quality control of endoscopic procedures. However, no relevant research has been reported. In this study, we aimed to develop a computer-assisted system, EndoAdd, for automated endoscopic surgical video analysis based on our dataset of endoscopic instrument images. METHODS Large training and validation datasets containing 45,143 images of 10 different endoscopic instruments and a test dataset of 18,375 images collected from several medical centers were used in this research. Annotated image frames were used to train the state-of-the-art object detection model, YOLO-v5, to identify the instruments. Based on the frame-level prediction results, we further developed a hidden Markov model to perform video analysis and generate heatmaps to summarize the videos. RESULTS EndoAdd achieved high accuracy (>97%) on the test dataset for all 10 endoscopic instrument types. The mean average accuracy, precision, recall, and F1-score were 99.1%, 92.0%, 88.8%, and 89.3%, respectively. The area under the curve values exceeded 0.94 for all instrument types. Heatmaps of endoscopic procedures were generated for both retrospective and real-time analyses. CONCLUSIONS We successfully developed an automated endoscopic video analysis system, EndoAdd, which supports retrospective assessment and real-time monitoring. It can be used for data analysis and quality control of endoscopic procedures in clinical practice.
Collapse
Affiliation(s)
- Yan Zhu
- Endoscopy Center and Endoscopy Research Institute, Zhongshan Hospital, Fudan University, Shanghai 200032, China; (Y.Z.); (L.D.); (P.-Y.F.); (Z.-H.G.); (D.-F.Z.); (W.-F.C.)
- Shanghai Collaborative Innovation Center of Endoscopy, Shanghai 200032, China
| | - Ling Du
- Endoscopy Center and Endoscopy Research Institute, Zhongshan Hospital, Fudan University, Shanghai 200032, China; (Y.Z.); (L.D.); (P.-Y.F.); (Z.-H.G.); (D.-F.Z.); (W.-F.C.)
- Shanghai Collaborative Innovation Center of Endoscopy, Shanghai 200032, China
| | - Pei-Yao Fu
- Endoscopy Center and Endoscopy Research Institute, Zhongshan Hospital, Fudan University, Shanghai 200032, China; (Y.Z.); (L.D.); (P.-Y.F.); (Z.-H.G.); (D.-F.Z.); (W.-F.C.)
- Shanghai Collaborative Innovation Center of Endoscopy, Shanghai 200032, China
| | - Zi-Han Geng
- Endoscopy Center and Endoscopy Research Institute, Zhongshan Hospital, Fudan University, Shanghai 200032, China; (Y.Z.); (L.D.); (P.-Y.F.); (Z.-H.G.); (D.-F.Z.); (W.-F.C.)
- Shanghai Collaborative Innovation Center of Endoscopy, Shanghai 200032, China
| | - Dan-Feng Zhang
- Endoscopy Center and Endoscopy Research Institute, Zhongshan Hospital, Fudan University, Shanghai 200032, China; (Y.Z.); (L.D.); (P.-Y.F.); (Z.-H.G.); (D.-F.Z.); (W.-F.C.)
- Shanghai Collaborative Innovation Center of Endoscopy, Shanghai 200032, China
| | - Wei-Feng Chen
- Endoscopy Center and Endoscopy Research Institute, Zhongshan Hospital, Fudan University, Shanghai 200032, China; (Y.Z.); (L.D.); (P.-Y.F.); (Z.-H.G.); (D.-F.Z.); (W.-F.C.)
- Shanghai Collaborative Innovation Center of Endoscopy, Shanghai 200032, China
| | - Quan-Lin Li
- Endoscopy Center and Endoscopy Research Institute, Zhongshan Hospital, Fudan University, Shanghai 200032, China; (Y.Z.); (L.D.); (P.-Y.F.); (Z.-H.G.); (D.-F.Z.); (W.-F.C.)
- Shanghai Collaborative Innovation Center of Endoscopy, Shanghai 200032, China
| | - Ping-Hong Zhou
- Endoscopy Center and Endoscopy Research Institute, Zhongshan Hospital, Fudan University, Shanghai 200032, China; (Y.Z.); (L.D.); (P.-Y.F.); (Z.-H.G.); (D.-F.Z.); (W.-F.C.)
- Shanghai Collaborative Innovation Center of Endoscopy, Shanghai 200032, China
| |
Collapse
|
6
|
Kang YJ, Kim SJ, Seo SH, Lee S, Kim HS, Yoo JI. Assessment of Automated Identification of Phases in Videos of Total Hip Arthroplasty Using Deep Learning Techniques. Clin Orthop Surg 2024; 16:210-216. [PMID: 38562629 PMCID: PMC10973629 DOI: 10.4055/cios23280] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/12/2023] [Revised: 10/23/2023] [Accepted: 11/06/2023] [Indexed: 04/04/2024] Open
Abstract
Background As the population ages, the rates of hip diseases and fragility fractures are increasing, making total hip arthroplasty (THA) one of the best methods for treating elderly patients. With the increasing number of THA surgeries and diverse surgical methods, there is a need for standard evaluation protocols. This study aimed to use deep learning algorithms to classify THA videos and evaluate the accuracy of the labelling of these videos. Methods In our study, we manually annotated 7 phases in THA, including skin incision, broaching, exposure of acetabulum, acetabular reaming, acetabular cup positioning, femoral stem insertion, and skin closure. Within each phase, a second trained annotator marked the beginning and end of instrument usages, such as the skin blade, forceps, Bovie, suction device, suture material, retractor, rasp, femoral stem, acetabular reamer, head trial, and real head. Results In our study, we utilized YOLOv3 to collect 540 operating images of THA procedures and create a scene annotation model. The results of our study showed relatively high accuracy in the clear classification of surgical techniques such as skin incision and closure, broaching, acetabular reaming, and femoral stem insertion, with a mean average precision (mAP) of 0.75 or higher. Most of the equipment showed good accuracy of mAP 0.7 or higher, except for the suction device, suture material, and retractor. Conclusions Scene annotation for the instrument and phases in THA using deep learning techniques may provide potentially useful tools for subsequent documentation, assessment of skills, and feedback.
Collapse
Affiliation(s)
- Yang Jae Kang
- Division of Bio and Medical Big Data Department (BK4 Program) and Life Science Department, Gyeongsang National University, Jinju, Korea
| | - Shin June Kim
- Biomedical Research Institute, Inha University Hospital, Incheon, Korea
| | - Sung Hyo Seo
- Biomedical Research Institute, Gyeongsang National University Hospital, Jinju, Korea
| | - Sangyeob Lee
- Biomedical Research Institute, Gyeongsang National University Hospital, Jinju, Korea
| | - Hyeon Su Kim
- Biomedical Research Institute, Inha University Hospital, Incheon, Korea
| | - Jun-Il Yoo
- Department of Orthopedic Surgery, Inha University Hospital, Inha University College of Medicine, Incheon, Korea
| |
Collapse
|
7
|
Smithmaitrie P, Khaonualsri M, Sae-Lim W, Wangkulangkul P, Jearanai S, Cheewatanakornkul S. Development of deep learning framework for anatomical landmark detection and guided dissection line during laparoscopic cholecystectomy. Heliyon 2024; 10:e25210. [PMID: 38327394 PMCID: PMC10847946 DOI: 10.1016/j.heliyon.2024.e25210] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/18/2022] [Revised: 11/15/2023] [Accepted: 01/23/2024] [Indexed: 02/09/2024] Open
Abstract
Background Bile duct injuries during laparoscopic cholecystectomy can arise from misinterpretation of biliary anatomy, leading to dissection in improper areas. The integration of a deep learning framework into laparoscopic procedures offers the potential for real-time anatomical landmark recognition, ensuring accurate dissection. The objective of this study is to develop a deep learning framework that can precisely identify anatomical landmarks, including Rouviere's sulcus and the liver base of segment IV, and provide a guided dissection line during laparoscopic cholecystectomy. Methods We retrospectively collected 40 laparoscopic cholecystectomy videos and extracted 80 images form each video to establish the dataset. Three surgeons annotated the bounding boxes of anatomical landmarks on a total of 3200 images. The YOLOv7 model was trained to detect Rouviere's sulcus and the liver base of segment IV as anatomical landmarks. Additionally, the guided dissection line was generated between these two landmarks by the proposed algorithm. To evaluate the performance of the detection model, mean average precision (mAP), precision, and recall were calculated. Furthermore, the accuracy of the guided dissection line was evaluated by three surgeons. The performance of the detection model was compared to the scaled-YOLOv4 and YOLOv5 models. Finally, the proposed framework was deployed in the operating room for real-time detection and visualization. Results The overall performance of the YOLOv7 model on validation set and testing set were 98.1 % and 91.3 %, respectively. Surgeons accepted the visualization of guide dissection line with a rate of 95.71 %. In the operating room, the well-trained model accurately identified the anatomical landmarks and generated the guided dissection line in real-time. Conclusions The proposed framework effectively identifies anatomical landmarks and generates a guided dissection line in real-time during laparoscopic cholecystectomy. This research underscores the potential of using deep learning models as computer-assisted tools in surgery, providing an assistant tool to accommodate with surgeons.
Collapse
Affiliation(s)
- Pruittikorn Smithmaitrie
- Department of Mechanical and Mechatronics Engineering, Faculty of Engineering, Prince of Songkla University, Thailand
| | - Methasit Khaonualsri
- Department of Mechanical and Mechatronics Engineering, Faculty of Engineering, Prince of Songkla University, Thailand
| | - Wannipa Sae-Lim
- Department of Computer Science, Faculty of Science, Prince of Songkla University, Thailand
| | - Piyanun Wangkulangkul
- Minimally Invasive Surgery Unit, Department of Surgery, Faculty of Medicine, Prince of Songkla University, Thailand
| | - Supakool Jearanai
- Minimally Invasive Surgery Unit, Department of Surgery, Faculty of Medicine, Prince of Songkla University, Thailand
| | - Siripong Cheewatanakornkul
- Minimally Invasive Surgery Unit, Department of Surgery, Faculty of Medicine, Prince of Songkla University, Thailand
| |
Collapse
|
8
|
Komatsu M, Kitaguchi D, Yura M, Takeshita N, Yoshida M, Yamaguchi M, Kondo H, Kinoshita T, Ito M. Automatic surgical phase recognition-based skill assessment in laparoscopic distal gastrectomy using multicenter videos. Gastric Cancer 2024; 27:187-196. [PMID: 38038811 DOI: 10.1007/s10120-023-01450-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/20/2023] [Accepted: 10/31/2023] [Indexed: 12/02/2023]
Abstract
BACKGROUND Gastric surgery involves numerous surgical phases; however, its steps can be clearly defined. Deep learning-based surgical phase recognition can promote stylization of gastric surgery with applications in automatic surgical skill assessment. This study aimed to develop a deep learning-based surgical phase-recognition model using multicenter videos of laparoscopic distal gastrectomy, and examine the feasibility of automatic surgical skill assessment using the developed model. METHODS Surgical videos from 20 hospitals were used. Laparoscopic distal gastrectomy was defined and annotated into nine phases and a deep learning-based image classification model was developed for phase recognition. We examined whether the developed model's output, including the number of frames in each phase and the adequacy of the surgical field development during the phase of supra-pancreatic lymphadenectomy, correlated with the manually assigned skill assessment score. RESULTS The overall accuracy of phase recognition was 88.8%. Regarding surgical skill assessment based on the number of frames during the phases of lymphadenectomy of the left greater curvature and reconstruction, the number of frames in the high-score group were significantly less than those in the low-score group (829 vs. 1,152, P < 0.01; 1,208 vs. 1,586, P = 0.01, respectively). The output score of the adequacy of the surgical field development, which is the developed model's output, was significantly higher in the high-score group than that in the low-score group (0.975 vs. 0.970, P = 0.04). CONCLUSION The developed model had high accuracy in phase-recognition tasks and has the potential for application in automatic surgical skill assessment systems.
Collapse
Affiliation(s)
- Masaru Komatsu
- Gastric Surgery Division, National Cancer Center Hospital East, 6-5-1 Kashiwanoha, Kashiwa, Chiba, 277-8577, Japan
- Department for the Promotion of Medical Device Innovation, National Cancer Center Hospital East, 6-5-1 Kashiwanoha, Kashiwa, Chiba, 277-8577, Japan
- Course of Advanced Clinical Research of Cancer, Juntendo University Graduate School of Medicine, 2-1-1, Hongo, Bunkyo-Ward, Tokyo, 113-8421, Japan
| | - Daichi Kitaguchi
- Department for the Promotion of Medical Device Innovation, National Cancer Center Hospital East, 6-5-1 Kashiwanoha, Kashiwa, Chiba, 277-8577, Japan
| | - Masahiro Yura
- Gastric Surgery Division, National Cancer Center Hospital East, 6-5-1 Kashiwanoha, Kashiwa, Chiba, 277-8577, Japan
| | - Nobuyoshi Takeshita
- Department for the Promotion of Medical Device Innovation, National Cancer Center Hospital East, 6-5-1 Kashiwanoha, Kashiwa, Chiba, 277-8577, Japan
| | - Mitsumasa Yoshida
- Gastric Surgery Division, National Cancer Center Hospital East, 6-5-1 Kashiwanoha, Kashiwa, Chiba, 277-8577, Japan
| | - Masayuki Yamaguchi
- Course of Advanced Clinical Research of Cancer, Juntendo University Graduate School of Medicine, 2-1-1, Hongo, Bunkyo-Ward, Tokyo, 113-8421, Japan
| | - Hibiki Kondo
- Department for the Promotion of Medical Device Innovation, National Cancer Center Hospital East, 6-5-1 Kashiwanoha, Kashiwa, Chiba, 277-8577, Japan
| | - Takahiro Kinoshita
- Gastric Surgery Division, National Cancer Center Hospital East, 6-5-1 Kashiwanoha, Kashiwa, Chiba, 277-8577, Japan
| | - Masaaki Ito
- Department for the Promotion of Medical Device Innovation, National Cancer Center Hospital East, 6-5-1 Kashiwanoha, Kashiwa, Chiba, 277-8577, Japan.
- Surgical Device Innovation Office, National Cancer Center Hospital East, 6-5-1, Kashiwanoha, Kashiwa, Chiba, 277-8577, Japan.
| |
Collapse
|
9
|
Hegde SR, Namazi B, Iyengar N, Cao S, Desir A, Marques C, Mahnken H, Dumas RP, Sankaranarayanan G. Automated segmentation of phases, steps, and tasks in laparoscopic cholecystectomy using deep learning. Surg Endosc 2024; 38:158-170. [PMID: 37945709 DOI: 10.1007/s00464-023-10482-3] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/23/2023] [Accepted: 09/17/2023] [Indexed: 11/12/2023]
Abstract
BACKGROUND Video-based review is paramount for operative performance assessment but can be laborious when performed manually. Hierarchical Task Analysis (HTA) is a well-known method that divides any procedure into phases, steps, and tasks. HTA requires large datasets of videos with consistent definitions at each level. Our aim was to develop an AI model for automated segmentation of phases, steps, and tasks for laparoscopic cholecystectomy videos using a standardized HTA. METHODS A total of 160 laparoscopic cholecystectomy videos were collected from a publicly available dataset known as cholec80 and from our own institution. All videos were annotated for the beginning and ending of a predefined set of phases, steps, and tasks. Deep learning models were then separately developed and trained for the three levels using a 3D Convolutional Neural Network architecture. RESULTS Four phases, eight steps, and nineteen tasks were defined through expert consensus. The training set for our deep learning models contained 100 videos with an additional 20 videos for hyperparameter optimization and tuning. The remaining 40 videos were used for testing the performance. The overall accuracy for phases, steps, and tasks were 0.90, 0.81, and 0.65 with the average F1 score of 0.86, 0.76 and 0.48 respectively. Control of bleeding and bile spillage tasks were most variable in definition, operative management, and clinical relevance. CONCLUSION The use of hierarchical task analysis for surgical video analysis has numerous applications in AI-based automated systems. Our results show that our tiered method of task analysis can successfully be used to train a DL model.
Collapse
Affiliation(s)
- Shruti R Hegde
- Department of Surgery, University of Texas Southwestern Medical Center, 5323 Harry Hines Blvd., Dallas, TX, 75390-9159, USA
| | - Babak Namazi
- Department of Surgery, University of Texas Southwestern Medical Center, 5323 Harry Hines Blvd., Dallas, TX, 75390-9159, USA
| | - Niyenth Iyengar
- Department of Surgery, University of Texas Southwestern Medical Center, 5323 Harry Hines Blvd., Dallas, TX, 75390-9159, USA
| | - Sarah Cao
- Department of Surgery, University of Texas Southwestern Medical Center, 5323 Harry Hines Blvd., Dallas, TX, 75390-9159, USA
| | - Alexis Desir
- Department of Surgery, University of Texas Southwestern Medical Center, 5323 Harry Hines Blvd., Dallas, TX, 75390-9159, USA
| | - Carolina Marques
- Department of Surgery, University of Texas Southwestern Medical Center, 5323 Harry Hines Blvd., Dallas, TX, 75390-9159, USA
| | - Heidi Mahnken
- Department of Surgery, University of Texas Southwestern Medical Center, 5323 Harry Hines Blvd., Dallas, TX, 75390-9159, USA
| | - Ryan P Dumas
- Department of Surgery, University of Texas Southwestern Medical Center, 5323 Harry Hines Blvd., Dallas, TX, 75390-9159, USA
| | - Ganesh Sankaranarayanan
- Department of Surgery, University of Texas Southwestern Medical Center, 5323 Harry Hines Blvd., Dallas, TX, 75390-9159, USA.
| |
Collapse
|
10
|
Demir KC, Schieber H, Weise T, Roth D, May M, Maier A, Yang SH. Deep Learning in Surgical Workflow Analysis: A Review of Phase and Step Recognition. IEEE J Biomed Health Inform 2023; 27:5405-5417. [PMID: 37665700 DOI: 10.1109/jbhi.2023.3311628] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 09/06/2023]
Abstract
OBJECTIVE In the last two decades, there has been a growing interest in exploring surgical procedures with statistical models to analyze operations at different semantic levels. This information is necessary for developing context-aware intelligent systems, which can assist the physicians during operations, evaluate procedures afterward or help the management team to effectively utilize the operating room. The objective is to extract reliable patterns from surgical data for the robust estimation of surgical activities performed during operations. The purpose of this article is to review the state-of-the-art deep learning methods that have been published after 2018 for analyzing surgical workflows, with a focus on phase and step recognition. METHODS Three databases, IEEE Xplore, Scopus, and PubMed were searched, and additional studies are added through a manual search. After the database search, 343 studies were screened and a total of 44 studies are selected for this review. CONCLUSION The use of temporal information is essential for identifying the next surgical action. Contemporary methods used mainly RNNs, hierarchical CNNs, and Transformers to preserve long-distance temporal relations. The lack of large publicly available datasets for various procedures is a great challenge for the development of new and robust models. As supervised learning strategies are used to show proof-of-concept, self-supervised, semi-supervised, or active learning methods are used to mitigate dependency on annotated data. SIGNIFICANCE The present study provides a comprehensive review of recent methods in surgical workflow analysis, summarizes commonly used architectures, datasets, and discusses challenges.
Collapse
|
11
|
Kawamura M, Endo Y, Fujinaga A, Orimoto H, Amano S, Kawasaki T, Kawano Y, Masuda T, Hirashita T, Kimura M, Ejima A, Matsunobu Y, Shinozuka K, Tokuyasu T, Inomata M. Development of an artificial intelligence system for real-time intraoperative assessment of the Critical View of Safety in laparoscopic cholecystectomy. Surg Endosc 2023; 37:8755-8763. [PMID: 37567981 DOI: 10.1007/s00464-023-10328-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/26/2023] [Accepted: 07/19/2023] [Indexed: 08/13/2023]
Abstract
BACKGROUND The Critical View of Safety (CVS) was proposed in 1995 to prevent bile duct injury during laparoscopic cholecystectomy (LC). The achievement of CVS was evaluated subjectively. This study aimed to develop an artificial intelligence (AI) system to evaluate CVS scores in LC. MATERIALS AND METHODS AI software was developed to evaluate the achievement of CVS using an algorithm for image classification based on a deep convolutional neural network. Short clips of hepatocystic triangle dissection were converted from 72 LC videos, and 23,793 images were labeled for training data. The learning models were examined using metrics commonly used in machine learning. RESULTS The mean values of precision, recall, F-measure, specificity, and overall accuracy for all the criteria of the best model were 0.971, 0.737, 0.832, 0.966, and 0.834, respectively. It took approximately 6 fps to obtain scores for a single image. CONCLUSIONS Using the AI system, we successfully evaluated the achievement of the CVS criteria using still images and videos of hepatocystic triangle dissection in LC. This encourages surgeons to be aware of CVS and is expected to improve surgical safety.
Collapse
Affiliation(s)
- Masahiro Kawamura
- Department of Gastroenterological and Pediatric Surgery, Faculty of Medicine, Oita University, Oita, Japan.
| | - Yuichi Endo
- Department of Gastroenterological and Pediatric Surgery, Faculty of Medicine, Oita University, Oita, Japan
| | - Atsuro Fujinaga
- Department of Gastroenterological and Pediatric Surgery, Faculty of Medicine, Oita University, Oita, Japan
| | - Hiroki Orimoto
- Department of Gastroenterological and Pediatric Surgery, Faculty of Medicine, Oita University, Oita, Japan
| | - Shota Amano
- Department of Gastroenterological and Pediatric Surgery, Faculty of Medicine, Oita University, Oita, Japan
| | - Takahide Kawasaki
- Department of Gastroenterological and Pediatric Surgery, Faculty of Medicine, Oita University, Oita, Japan
| | - Yoko Kawano
- Department of Gastroenterological and Pediatric Surgery, Faculty of Medicine, Oita University, Oita, Japan
| | - Takashi Masuda
- Department of Gastroenterological and Pediatric Surgery, Faculty of Medicine, Oita University, Oita, Japan
| | - Teijiro Hirashita
- Department of Gastroenterological and Pediatric Surgery, Faculty of Medicine, Oita University, Oita, Japan
| | - Misako Kimura
- Department of Information System and Engineering, Faculty of Information Engineering, Fukuoka Institute of Technology, Fukuoka, Japan
| | - Aika Ejima
- Department of Information System and Engineering, Faculty of Information Engineering, Fukuoka Institute of Technology, Fukuoka, Japan
| | - Yusuke Matsunobu
- Department of Information System and Engineering, Faculty of Information Engineering, Fukuoka Institute of Technology, Fukuoka, Japan
| | - Ken'ichi Shinozuka
- Department of Information System and Engineering, Faculty of Information Engineering, Fukuoka Institute of Technology, Fukuoka, Japan
| | - Tatsushi Tokuyasu
- Department of Information System and Engineering, Faculty of Information Engineering, Fukuoka Institute of Technology, Fukuoka, Japan
| | - Masafumi Inomata
- Department of Gastroenterological and Pediatric Surgery, Faculty of Medicine, Oita University, Oita, Japan
| |
Collapse
|
12
|
Chen KA, Kirchoff KE, Butler LR, Holloway AD, Kapadia MR, Kuzmiak CM, Downs-Canner SM, Spanheimer PM, Gallagher KK, Gomez SM. Analysis of Specimen Mammography with Artificial Intelligence to Predict Margin Status. Ann Surg Oncol 2023; 30:7107-7115. [PMID: 37563337 PMCID: PMC10592216 DOI: 10.1245/s10434-023-14083-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/23/2023] [Accepted: 07/17/2023] [Indexed: 08/12/2023]
Abstract
BACKGROUND Intraoperative specimen mammography is a valuable tool in breast cancer surgery, providing immediate assessment of margins for a resected tumor. However, the accuracy of specimen mammography in detecting microscopic margin positivity is low. We sought to develop an artificial intelligence model to predict the pathologic margin status of resected breast tumors using specimen mammography. METHODS A dataset of specimen mammography images matched with pathologic margin status was collected from our institution from 2017 to 2020. The dataset was randomly split into training, validation, and test sets. Specimen mammography models pretrained on radiologic images were developed and compared with models pretrained on nonmedical images. Model performance was assessed using sensitivity, specificity, and area under the receiver operating characteristic curve (AUROC). RESULTS The dataset included 821 images, and 53% had positive margins. For three out of four model architectures tested, models pretrained on radiologic images outperformed nonmedical models. The highest performing model, InceptionV3, showed sensitivity of 84%, specificity of 42%, and AUROC of 0.71. Model performance was better among patients with invasive cancers, less dense breasts, and non-white race. CONCLUSIONS This study developed and internally validated artificial intelligence models that predict pathologic margins status for partial mastectomy from specimen mammograms. The models' accuracy compares favorably with published literature on surgeon and radiologist interpretation of specimen mammography. With further development, these models could more precisely guide the extent of resection, potentially improving cosmesis and reducing reoperations.
Collapse
Affiliation(s)
- Kevin A Chen
- Division of Surgical Oncology, Department of Surgery, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
| | - Kathryn E Kirchoff
- Joint Department of Biomedical Engineering, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
| | - Logan R Butler
- Division of Surgical Oncology, Department of Surgery, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
| | - Alexa D Holloway
- Division of Surgical Oncology, Department of Surgery, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
| | - Muneera R Kapadia
- Division of Surgical Oncology, Department of Surgery, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
| | - Cherie M Kuzmiak
- Department of Radiology, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
| | - Stephanie M Downs-Canner
- Department of Surgery, Breast Service, Memorial Sloan Kettering Cancer Center, New York, NY, USA
| | - Phillip M Spanheimer
- Division of Surgical Oncology, Department of Surgery, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
| | - Kristalyn K Gallagher
- Division of Surgical Oncology, Department of Surgery, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA.
| | - Shawn M Gomez
- Joint Department of Biomedical Engineering, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA.
| |
Collapse
|
13
|
Choksi S, Szot S, Zang C, Yarali K, Cao Y, Ahmad F, Xiang Z, Bitner DP, Kostic Z, Filicori F. Bringing Artificial Intelligence to the operating room: edge computing for real-time surgical phase recognition. Surg Endosc 2023; 37:8778-8784. [PMID: 37580578 DOI: 10.1007/s00464-023-10322-4] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/10/2023] [Accepted: 07/19/2023] [Indexed: 08/16/2023]
Abstract
BACKGROUND Automation of surgical phase recognition is a key effort toward the development of Computer Vision (CV) algorithms, for workflow optimization and video-based assessment. CV is a form of Artificial Intelligence (AI) that allows interpretation of images through a deep learning (DL)-based algorithm. The improvements in Graphic Processing Unit (GPU) computing devices allow researchers to apply these algorithms for recognition of content in videos in real-time. Edge computing, where data is collected, analyzed, and acted upon in close proximity to the collection source, is essential meet the demands of workflow optimization by providing real-time algorithm application. We implemented a real-time phase recognition workflow and demonstrated its performance on 10 Robotic Inguinal Hernia Repairs (RIHR) to obtain phase predictions during the procedure. METHODS Our phase recognition algorithm was developed with 211 videos of RIHR originally annotated into 14 surgical phases. Using these videos, a DL model with a ResNet-50 backbone was trained and validated to automatically recognize surgical phases. The model was deployed to a GPU, the Nvidia® Jetson Xavier™ NX edge computing device. RESULTS This model was tested on 10 inguinal hernia repairs from four surgeons in real-time. The model was improved using post-recording processing methods such as phase merging into seven final phases (peritoneal scoring, mesh placement, preperitoneal dissection, reduction of hernia, out of body, peritoneal closure, and transitionary idle) and averaging of frames. Predictions were made once per second with a processing latency of approximately 250 ms. The accuracy of the real-time predictions ranged from 59.8 to 78.2% with an average accuracy of 68.7%. CONCLUSION A real-time phase prediction of RIHR using a CV deep learning model was successfully implemented. This real-time CV phase segmentation system can be useful for monitoring surgical progress and be integrated into software to provide hospital workflow optimization.
Collapse
Affiliation(s)
- Sarah Choksi
- Intraoperative Performance Analytics Laboratory (IPAL), Department of Surgery, Lenox Hill Hospital, Northwell Health, 186 E 76th Street, 1st Fl, New York, NY, 10021, USA.
| | - Skyler Szot
- Department of Electrical Engineering, Columbia University, 500 W 120 Street, Mudd 1310, New York, NY, 10027, USA
| | - Chengbo Zang
- Department of Electrical Engineering, Columbia University, 500 W 120 Street, Mudd 1310, New York, NY, 10027, USA
| | - Kaan Yarali
- Department of Electrical Engineering, Columbia University, 500 W 120 Street, Mudd 1310, New York, NY, 10027, USA
| | - Yuqing Cao
- Department of Electrical Engineering, Columbia University, 500 W 120 Street, Mudd 1310, New York, NY, 10027, USA
| | - Feroz Ahmad
- Department of Electrical Engineering, Columbia University, 500 W 120 Street, Mudd 1310, New York, NY, 10027, USA
| | - Zixuan Xiang
- Department of Electrical Engineering, Columbia University, 500 W 120 Street, Mudd 1310, New York, NY, 10027, USA
| | - Daniel P Bitner
- Intraoperative Performance Analytics Laboratory (IPAL), Department of Surgery, Lenox Hill Hospital, Northwell Health, 186 E 76th Street, 1st Fl, New York, NY, 10021, USA
| | - Zoran Kostic
- Department of Electrical Engineering, Columbia University, 500 W 120 Street, Mudd 1310, New York, NY, 10027, USA
| | - Filippo Filicori
- Intraoperative Performance Analytics Laboratory (IPAL), Department of Surgery, Lenox Hill Hospital, Northwell Health, 186 E 76th Street, 1st Fl, New York, NY, 10021, USA
- Zucker School of Medicine at Hofstra/Northwell Health, 5000 Hofstra Blvd, Hempstead, NY, 11549, USA
| |
Collapse
|
14
|
Graëff C, Daiss A, Lampert T, Padoy N, Martins A, Sapa MC, Liverneaux P. Preliminary stage in the development of an artificial intelligence algorithm: Variations between 100 surgeons in phase annotation in a video of internal fixation of distal radius fracture. Orthop Traumatol Surg Res 2023; 109:103564. [PMID: 36702298 DOI: 10.1016/j.otsr.2023.103564] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/11/2022] [Revised: 11/16/2022] [Accepted: 12/13/2022] [Indexed: 01/25/2023]
Abstract
INTRODUCTION In order to be used naturally and widely, an artificial intelligence algorithm of phase detection in surgical videos presupposes an expert consensus defining phases. OBJECTIVES The aim of the present study was to seek consensus in defining the various phases of a surgical technique in wrist traumatology. METHODS Three thousand two hundred and twenty-nine surgeons were sent a video showing anterior plate fixation of the distal radius and a questionnaire on the number of phases they distinguished and the visual cues signaling the beginning of each phase. Three experimenters predefined the number of phases (5: installation, approach, fixation, verification, closure) and sub-phases (3a: introduction of plate; 3b: positioning distal screws; 3c: positioning proximal screws) and the cues signaling the beginning of each. The numbers of the responses per item were collected. RESULTS Only 216 (6.7%) surgeons opened the questionnaire, and 100 answered all questions (3.1%). Most respondents claimed 5/5 expertise. Number of phases identified ranged between 3 and 10. More than two-thirds of respondents identified the same phase cue as defined by the 3 experimenters in most cases, except for "verification" and "positioning proximal screws". DISCUSSION Surgical procedures comprise a succession of phases, the beginning or end of which can be defined by a precise visual cue on video, either beginning with the appearance of the cue or the disappearance of the cue defining the preceding phase. CONCLUSION These cues need to be defined very precisely before attempting manual annotation of surgical videos in order to develop an artificial intelligence algorithm. LEVEL OF EVIDENCE II.
Collapse
Affiliation(s)
- Camille Graëff
- ICube CNRS UMR7357, Strasbourg University, 2-4, rue Boussingault, 67000 Strasbourg, France; IHU, Institute of image-guided surgery, Strasbourg, France
| | - Audrey Daiss
- Department of hand surgery, Strasbourg University Hospitals, FMTS, 1, avenue Molière, 67200 Strasbourg, France
| | - Thomas Lampert
- ICube CNRS UMR7357, Strasbourg University, 2-4, rue Boussingault, 67000 Strasbourg, France
| | - Nicolas Padoy
- ICube CNRS UMR7357, Strasbourg University, 2-4, rue Boussingault, 67000 Strasbourg, France; IHU, Institute of image-guided surgery, Strasbourg, France
| | - Antoine Martins
- Department of hand surgery, Strasbourg University Hospitals, FMTS, 1, avenue Molière, 67200 Strasbourg, France
| | - Marie-Cécile Sapa
- Department of hand surgery, Strasbourg University Hospitals, FMTS, 1, avenue Molière, 67200 Strasbourg, France
| | - Philippe Liverneaux
- ICube CNRS UMR7357, Strasbourg University, 2-4, rue Boussingault, 67000 Strasbourg, France; Department of hand surgery, Strasbourg University Hospitals, FMTS, 1, avenue Molière, 67200 Strasbourg, France.
| |
Collapse
|
15
|
Gholinejad M, Edwin B, Elle OJ, Dankelman J, Loeve AJ. Process model analysis of parenchyma sparing laparoscopic liver surgery to recognize surgical steps and predict impact of new technologies. Surg Endosc 2023; 37:7083-7099. [PMID: 37386254 PMCID: PMC10462556 DOI: 10.1007/s00464-023-10166-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/15/2022] [Accepted: 05/28/2023] [Indexed: 07/01/2023]
Abstract
BACKGROUND Surgical process model (SPM) analysis is a great means to predict the surgical steps in a procedure as well as to predict the potential impact of new technologies. Especially in complicated and high-volume treatments, such as parenchyma sparing laparoscopic liver resection (LLR), profound process knowledge is essential for enabling improving surgical quality and efficiency. METHODS Videos of thirteen parenchyma sparing LLR were analyzed to extract the duration and sequence of surgical steps according to the process model. The videos were categorized into three groups, based on the tumor locations. Next, a detailed discrete events simulation model (DESM) of LLR was built, based on the process model and the process data obtained from the endoscopic videos. Furthermore, the impact of using a navigation platform on the total duration of the LLR was studied with the simulation model by assessing three different scenarios: (i) no navigation platform, (ii) conservative positive effect, and (iii) optimistic positive effect. RESULTS The possible variations of sequences of surgical steps in performing parenchyma sparing depending on the tumor locations were established. The statistically most probable chain of surgical steps was predicted, which could be used to improve parenchyma sparing surgeries. In all three categories (i-iii) the treatment phase covered the major part (~ 40%) of the total procedure duration (bottleneck). The simulation results predict that a navigation platform could decrease the total surgery duration by up to 30%. CONCLUSION This study showed a DESM based on the analysis of steps during surgical procedures can be used to predict the impact of new technology. SPMs can be used to detect, e.g., the most probable workflow paths which enables predicting next surgical steps, improving surgical training systems, and analyzing surgical performance. Moreover, it provides insight into the points for improvement and bottlenecks in the surgical process.
Collapse
Affiliation(s)
- Maryam Gholinejad
- Department of Biomechanical Engineering, Faculty of Mechanical, Maritime and Materials Engineering, Delft University of Technology, Delft, The Netherlands.
| | - Bjørn Edwin
- The Intervention Centre, Oslo University Hospital, Oslo, Norway
- Medical Faculty, Institute of Clinical Medicine, University of Oslo, Oslo, Norway
- Department of HPB Surgery, Oslo University Hospital, Oslo, Norway
| | - Ole Jakob Elle
- The Intervention Centre, Oslo University Hospital, Oslo, Norway
- Department of Informatics, University of Oslo, Oslo, Norway
| | - Jenny Dankelman
- Department of Biomechanical Engineering, Faculty of Mechanical, Maritime and Materials Engineering, Delft University of Technology, Delft, The Netherlands
| | - Arjo J Loeve
- Department of Biomechanical Engineering, Faculty of Mechanical, Maritime and Materials Engineering, Delft University of Technology, Delft, The Netherlands
| |
Collapse
|
16
|
Igaki T, Kitaguchi D, Matsuzaki H, Nakajima K, Kojima S, Hasegawa H, Takeshita N, Kinugasa Y, Ito M. Automatic Surgical Skill Assessment System Based on Concordance of Standardized Surgical Field Development Using Artificial Intelligence. JAMA Surg 2023; 158:e231131. [PMID: 37285142 PMCID: PMC10248810 DOI: 10.1001/jamasurg.2023.1131] [Citation(s) in RCA: 9] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/15/2022] [Accepted: 01/28/2023] [Indexed: 06/08/2023]
Abstract
Importance Automatic surgical skill assessment with artificial intelligence (AI) is more objective than manual video review-based skill assessment and can reduce human burden. Standardization of surgical field development is an important aspect of this skill assessment. Objective To develop a deep learning model that can recognize the standardized surgical fields in laparoscopic sigmoid colon resection and to evaluate the feasibility of automatic surgical skill assessment based on the concordance of the standardized surgical field development using the proposed deep learning model. Design, Setting, and Participants This retrospective diagnostic study used intraoperative videos of laparoscopic colorectal surgery submitted to the Japan Society for Endoscopic Surgery between August 2016 and November 2017. Data were analyzed from April 2020 to September 2022. Interventions Videos of surgery performed by expert surgeons with Endoscopic Surgical Skill Qualification System (ESSQS) scores higher than 75 were used to construct a deep learning model able to recognize a standardized surgical field and output its similarity to standardized surgical field development as an AI confidence score (AICS). Other videos were extracted as the validation set. Main Outcomes and Measures Videos with scores less than or greater than 2 SDs from the mean were defined as the low- and high-score groups, respectively. The correlation between AICS and ESSQS score and the screening performance using AICS for low- and high-score groups were analyzed. Results The sample included 650 intraoperative videos, 60 of which were used for model construction and 60 for validation. The Spearman rank correlation coefficient between the AICS and ESSQS score was 0.81. The receiver operating characteristic (ROC) curves for the screening of the low- and high-score groups were plotted, and the areas under the ROC curve for the low- and high-score group screening were 0.93 and 0.94, respectively. Conclusions and Relevance The AICS from the developed model strongly correlated with the ESSQS score, demonstrating the model's feasibility for use as a method of automatic surgical skill assessment. The findings also suggest the feasibility of the proposed model for creating an automated screening system for surgical skills and its potential application to other types of endoscopic procedures.
Collapse
Affiliation(s)
- Takahiro Igaki
- Surgical Device Innovation Office, National Cancer Center Hospital East, Kashiwanoha, Kashiwa, Chiba, Japan
- Department of Colorectal Surgery, National Cancer Center Hospital East, Kashiwanoha, Kashiwa, Chiba, Japan
- Department of Gastrointestinal Surgery, Tokyo Medical and Dental University Graduate School of Medicine, Yushima, Bunkyo-Ku, Tokyo, Japan
| | - Daichi Kitaguchi
- Surgical Device Innovation Office, National Cancer Center Hospital East, Kashiwanoha, Kashiwa, Chiba, Japan
- Department of Colorectal Surgery, National Cancer Center Hospital East, Kashiwanoha, Kashiwa, Chiba, Japan
| | - Hiroki Matsuzaki
- Surgical Device Innovation Office, National Cancer Center Hospital East, Kashiwanoha, Kashiwa, Chiba, Japan
| | - Kei Nakajima
- Surgical Device Innovation Office, National Cancer Center Hospital East, Kashiwanoha, Kashiwa, Chiba, Japan
- Department of Colorectal Surgery, National Cancer Center Hospital East, Kashiwanoha, Kashiwa, Chiba, Japan
| | - Shigehiro Kojima
- Surgical Device Innovation Office, National Cancer Center Hospital East, Kashiwanoha, Kashiwa, Chiba, Japan
- Department of Colorectal Surgery, National Cancer Center Hospital East, Kashiwanoha, Kashiwa, Chiba, Japan
| | - Hiro Hasegawa
- Surgical Device Innovation Office, National Cancer Center Hospital East, Kashiwanoha, Kashiwa, Chiba, Japan
- Department of Colorectal Surgery, National Cancer Center Hospital East, Kashiwanoha, Kashiwa, Chiba, Japan
| | - Nobuyoshi Takeshita
- Surgical Device Innovation Office, National Cancer Center Hospital East, Kashiwanoha, Kashiwa, Chiba, Japan
- Department of Colorectal Surgery, National Cancer Center Hospital East, Kashiwanoha, Kashiwa, Chiba, Japan
| | - Yusuke Kinugasa
- Department of Gastrointestinal Surgery, Tokyo Medical and Dental University Graduate School of Medicine, Yushima, Bunkyo-Ku, Tokyo, Japan
| | - Masaaki Ito
- Surgical Device Innovation Office, National Cancer Center Hospital East, Kashiwanoha, Kashiwa, Chiba, Japan
- Department of Colorectal Surgery, National Cancer Center Hospital East, Kashiwanoha, Kashiwa, Chiba, Japan
| |
Collapse
|
17
|
Spinelli A, Carrano FM, Laino ME, Andreozzi M, Koleth G, Hassan C, Repici A, Chand M, Savevski V, Pellino G. Artificial intelligence in colorectal surgery: an AI-powered systematic review. Tech Coloproctol 2023; 27:615-629. [PMID: 36805890 DOI: 10.1007/s10151-023-02772-8] [Citation(s) in RCA: 7] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/14/2022] [Accepted: 02/07/2023] [Indexed: 02/23/2023]
Abstract
Artificial intelligence (AI) has the potential to revolutionize surgery in the coming years. Still, it is essential to clarify what the meaningful current applications are and what can be reasonably expected. This AI-powered review assessed the role of AI in colorectal surgery. A Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA)-compliant systematic search of PubMed, Embase, Scopus, Cochrane Library databases, and gray literature was conducted on all available articles on AI in colorectal surgery (from January 1 1997 to March 1 2021), aiming to define the perioperative applications of AI. Potentially eligible studies were identified using novel software powered by natural language processing (NLP) and machine learning (ML) technologies dedicated to systematic reviews. Out of 1238 articles identified, 115 were included in the final analysis. Available articles addressed the role of AI in several areas of interest. In the preoperative phase, AI can be used to define tailored treatment algorithms, support clinical decision-making, assess the risk of complications, and predict surgical outcomes and survival. Intraoperatively, AI-enhanced surgery and integration of AI in robotic platforms have been suggested. After surgery, AI can be implemented in the Enhanced Recovery after Surgery (ERAS) pathway. Additional areas of applications included the assessment of patient-reported outcomes, automated pathology assessment, and research. Available data on these aspects are limited, and AI in colorectal surgery is still in its infancy. However, the rapid evolution of technologies makes it likely that it will increasingly be incorporated into everyday practice.
Collapse
Affiliation(s)
- A Spinelli
- IRCCS Humanitas Research Hospital, via Manzoni 56, 20089, Rozzano, MI, Italy.
- Department of Biomedical Sciences, Humanitas University, Via Rita Levi Montalcini 4, 20090, Pieve Emanuele, MI, Italy.
| | - F M Carrano
- IRCCS Humanitas Research Hospital, via Manzoni 56, 20089, Rozzano, MI, Italy
| | - M E Laino
- Artificial Intelligence Center, Humanitas Clinical and Research Center-IRCCS, Via A. Manzoni 56, 20089, Rozzano, MI, Italy
| | - M Andreozzi
- Department of Clinical Medicine and Surgery, University "Federico II" of Naples, Naples, Italy
| | - G Koleth
- Department of Gastroenterology and Hepatology, Hospital Selayang, Selangor, Malaysia
| | - C Hassan
- IRCCS Humanitas Research Hospital, via Manzoni 56, 20089, Rozzano, MI, Italy
| | - A Repici
- IRCCS Humanitas Research Hospital, via Manzoni 56, 20089, Rozzano, MI, Italy
| | - M Chand
- Wellcome EPSRC Centre for Interventional and Surgical Sciences (WEISS), University College London, London, UK
| | - V Savevski
- Artificial Intelligence Center, Humanitas Clinical and Research Center-IRCCS, Via A. Manzoni 56, 20089, Rozzano, MI, Italy
| | - G Pellino
- Department of Advanced Medical and Surgical Sciences, Università degli Studi della Campania "Luigi Vanvitelli", Naples, Italy
- Colorectal Surgery, Vall d'Hebron University Hospital, Universitat Autonoma de Barcelona UAB, Barcelona, Spain
| |
Collapse
|
18
|
Kolbinger FR, Bodenstedt S, Carstens M, Leger S, Krell S, Rinner FM, Nielen TP, Kirchberg J, Fritzmann J, Weitz J, Distler M, Speidel S. Artificial Intelligence for context-aware surgical guidance in complex robot-assisted oncological procedures: An exploratory feasibility study. EUROPEAN JOURNAL OF SURGICAL ONCOLOGY 2023:106996. [PMID: 37591704 DOI: 10.1016/j.ejso.2023.106996] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/13/2023] [Revised: 06/20/2023] [Accepted: 07/26/2023] [Indexed: 08/19/2023]
Abstract
INTRODUCTION Complex oncological procedures pose various surgical challenges including dissection in distinct tissue planes and preservation of vulnerable anatomical structures throughout different surgical phases. In rectal surgery, violation of dissection planes increases the risk of local recurrence and autonomous nerve damage resulting in incontinence and sexual dysfunction. This work explores the feasibility of phase recognition and target structure segmentation in robot-assisted rectal resection (RARR) using machine learning. MATERIALS AND METHODS A total of 57 RARR were recorded and subsets of these were annotated with respect to surgical phases and exact locations of target structures (anatomical structures, tissue types, static structures, and dissection areas). For surgical phase recognition, three machine learning models were trained: LSTM, MSTCN, and Trans-SVNet. Based on pixel-wise annotations of target structures in 9037 images, individual segmentation models based on DeepLabv3 were trained. Model performance was evaluated using F1 score, Intersection-over-Union (IoU), accuracy, precision, recall, and specificity. RESULTS The best results for phase recognition were achieved with the MSTCN model (F1 score: 0.82 ± 0.01, accuracy: 0.84 ± 0.03). Mean IoUs for target structure segmentation ranged from 0.14 ± 0.22 to 0.80 ± 0.14 for organs and tissue types and from 0.11 ± 0.11 to 0.44 ± 0.30 for dissection areas. Image quality, distorting factors (i.e. blood, smoke), and technical challenges (i.e. lack of depth perception) considerably impacted segmentation performance. CONCLUSION Machine learning-based phase recognition and segmentation of selected target structures are feasible in RARR. In the future, such functionalities could be integrated into a context-aware surgical guidance system for rectal surgery.
Collapse
Affiliation(s)
- Fiona R Kolbinger
- Department of Visceral, Thoracic and Vascular Surgery, University Hospital and Faculty of Medicine Carl Gustav Carus, Technische Universität Dresden, Fetscherstraße 74, 01307 Dresden, Germany; National Center for Tumor Diseases Dresden (NCT/UCC), Germany: German Cancer Research Center (DKFZ), Heidelberg, Germany; Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany; Helmholtz-Zentrum Dresden - Rossendorf, Dresden, Germany; Else Kröner Fresenius Center for Digital Health (EKFZ), Technische Universität Dresden, Fetscherstraße 74, 01307, Dresden, Germany.
| | - Sebastian Bodenstedt
- Department of Translational Surgical Oncology, National Center for Tumor Diseases (NCT/UCC), Partner Site Dresden, Fetscherstraße 74, 01307, Dresden, Germany; Centre for Tactile Internet with Human-in-the-Loop (CeTI), Technische Universität Dresden, Dresden, Germany
| | - Matthias Carstens
- Department of Visceral, Thoracic and Vascular Surgery, University Hospital and Faculty of Medicine Carl Gustav Carus, Technische Universität Dresden, Fetscherstraße 74, 01307 Dresden, Germany
| | - Stefan Leger
- Else Kröner Fresenius Center for Digital Health (EKFZ), Technische Universität Dresden, Fetscherstraße 74, 01307, Dresden, Germany; Department of Translational Surgical Oncology, National Center for Tumor Diseases (NCT/UCC), Partner Site Dresden, Fetscherstraße 74, 01307, Dresden, Germany
| | - Stefanie Krell
- Department of Translational Surgical Oncology, National Center for Tumor Diseases (NCT/UCC), Partner Site Dresden, Fetscherstraße 74, 01307, Dresden, Germany
| | - Franziska M Rinner
- Department of Visceral, Thoracic and Vascular Surgery, University Hospital and Faculty of Medicine Carl Gustav Carus, Technische Universität Dresden, Fetscherstraße 74, 01307 Dresden, Germany
| | - Thomas P Nielen
- Department of Visceral, Thoracic and Vascular Surgery, University Hospital and Faculty of Medicine Carl Gustav Carus, Technische Universität Dresden, Fetscherstraße 74, 01307 Dresden, Germany
| | - Johanna Kirchberg
- Department of Visceral, Thoracic and Vascular Surgery, University Hospital and Faculty of Medicine Carl Gustav Carus, Technische Universität Dresden, Fetscherstraße 74, 01307 Dresden, Germany; National Center for Tumor Diseases Dresden (NCT/UCC), Germany: German Cancer Research Center (DKFZ), Heidelberg, Germany; Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany; Helmholtz-Zentrum Dresden - Rossendorf, Dresden, Germany
| | - Johannes Fritzmann
- Department of Visceral, Thoracic and Vascular Surgery, University Hospital and Faculty of Medicine Carl Gustav Carus, Technische Universität Dresden, Fetscherstraße 74, 01307 Dresden, Germany; National Center for Tumor Diseases Dresden (NCT/UCC), Germany: German Cancer Research Center (DKFZ), Heidelberg, Germany; Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany; Helmholtz-Zentrum Dresden - Rossendorf, Dresden, Germany
| | - Jürgen Weitz
- Department of Visceral, Thoracic and Vascular Surgery, University Hospital and Faculty of Medicine Carl Gustav Carus, Technische Universität Dresden, Fetscherstraße 74, 01307 Dresden, Germany; National Center for Tumor Diseases Dresden (NCT/UCC), Germany: German Cancer Research Center (DKFZ), Heidelberg, Germany; Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany; Helmholtz-Zentrum Dresden - Rossendorf, Dresden, Germany; Else Kröner Fresenius Center for Digital Health (EKFZ), Technische Universität Dresden, Fetscherstraße 74, 01307, Dresden, Germany; Centre for Tactile Internet with Human-in-the-Loop (CeTI), Technische Universität Dresden, Dresden, Germany
| | - Marius Distler
- Department of Visceral, Thoracic and Vascular Surgery, University Hospital and Faculty of Medicine Carl Gustav Carus, Technische Universität Dresden, Fetscherstraße 74, 01307 Dresden, Germany; National Center for Tumor Diseases Dresden (NCT/UCC), Germany: German Cancer Research Center (DKFZ), Heidelberg, Germany; Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany; Helmholtz-Zentrum Dresden - Rossendorf, Dresden, Germany
| | - Stefanie Speidel
- Else Kröner Fresenius Center for Digital Health (EKFZ), Technische Universität Dresden, Fetscherstraße 74, 01307, Dresden, Germany; Department of Translational Surgical Oncology, National Center for Tumor Diseases (NCT/UCC), Partner Site Dresden, Fetscherstraße 74, 01307, Dresden, Germany; Centre for Tactile Internet with Human-in-the-Loop (CeTI), Technische Universität Dresden, Dresden, Germany.
| |
Collapse
|
19
|
Bos J, Kundrat D, Dagnino G. Towards an Action Recognition Framework for Endovascular Surgery. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2023; 2023:1-5. [PMID: 38083619 DOI: 10.1109/embc40787.2023.10341057] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/18/2023]
Abstract
Objective knowledge about instrument manoeuvres in endovascular surgery is essential for evaluating surgical skills and developing advanced technologies for cathlab routines. To the recent day, endovascular navigation has been exclusively assessed in laboratory scenarios. By contrast, information contained in available fluoroscopy data from clinical cases has been disregarded. In this work, we pioneer a learning-based framework for motion activity recognition in fluoroscopy sequences. The architecture is composed of two networks for instrument segmentation and action recognition. In this preliminary study, we demonstrate feasibility of recognising instrument manoeuvres automatically in our ex vivo datasets.Clinical relevance-The proposed framework contributes to image-based and automated assessment of endovascular tasks. This facilitates robotic control development, surgical education, and smart clinical documentation.
Collapse
|
20
|
Kinoshita T, Komatsu M. Artificial Intelligence in Surgery and Its Potential for Gastric Cancer. J Gastric Cancer 2023; 23:400-409. [PMID: 37553128 PMCID: PMC10412972 DOI: 10.5230/jgc.2023.23.e27] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/15/2023] [Revised: 07/19/2023] [Accepted: 07/20/2023] [Indexed: 08/10/2023] Open
Abstract
Artificial intelligence (AI) has made significant progress in recent years, and many medical fields are attempting to introduce AI technology into clinical practice. Currently, much research is being conducted to evaluate that AI can be incorporated into surgical procedures to make them safer and more efficient, subsequently to obtain better outcomes for patients. In this paper, we review basic AI research regarding surgery and discuss the potential for implementing AI technology in gastric cancer surgery. At present, research and development is focused on AI technologies that assist the surgeon's understandings and judgment during surgery, such as anatomical navigation. AI systems are also being developed to recognize in which the surgical phase is ongoing. Such a surgical phase recognition systems is considered for effective storage of surgical videos and education, in the future, for use in systems to objectively evaluate the skill of surgeons. At this time, it is not considered practical to let AI make intraoperative decisions or move forceps automatically from an ethical standpoint, too. At present, AI research on surgery has various limitations, and it is desirable to develop practical systems that will truly benefit clinical practice in the future.
Collapse
Affiliation(s)
- Takahiro Kinoshita
- Gastric Surgery Division, National Cancer Center Hospital East, Kashiwa, Japan.
| | - Masaru Komatsu
- Gastric Surgery Division, National Cancer Center Hospital East, Kashiwa, Japan
| |
Collapse
|
21
|
Sone K, Tanimoto S, Toyohara Y, Taguchi A, Miyamoto Y, Mori M, Iriyama T, Wada-Hiraike O, Osuga Y. Evolution of a surgical system using deep learning in minimally invasive surgery (Review). Biomed Rep 2023; 19:45. [PMID: 37324165 PMCID: PMC10265572 DOI: 10.3892/br.2023.1628] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/05/2022] [Accepted: 03/31/2023] [Indexed: 06/17/2023] Open
Abstract
Recently, artificial intelligence (AI) has been applied in various fields due to the development of new learning methods, such as deep learning, and the marked progress in computational processing speed. AI is also being applied in the medical field for medical image recognition and omics analysis of genomes and other data. Recently, AI applications for videos of minimally invasive surgeries have also advanced, and studies on such applications are increasing. In the present review, studies that focused on the following topics were selected: i) Organ and anatomy identification, ii) instrument identification, iii) procedure and surgical phase recognition, iv) surgery-time prediction, v) identification of an appropriate incision line, and vi) surgical education. The development of autonomous surgical robots is also progressing, with the Smart Tissue Autonomous Robot (STAR) and RAVEN systems being the most reported developments. STAR, in particular, is currently being used in laparoscopic imaging to recognize the surgical site from laparoscopic images and is in the process of establishing an automated suturing system, albeit in animal experiments. The present review examined the possibility of fully autonomous surgical robots in the future.
Collapse
Affiliation(s)
- Kenbun Sone
- Department of Obstetrics and Gynecology, Faculty of Medicine, The University of Tokyo, Tokyo 113-8655, Japan
| | - Saki Tanimoto
- Department of Obstetrics and Gynecology, Faculty of Medicine, The University of Tokyo, Tokyo 113-8655, Japan
| | - Yusuke Toyohara
- Department of Obstetrics and Gynecology, Faculty of Medicine, The University of Tokyo, Tokyo 113-8655, Japan
| | - Ayumi Taguchi
- Department of Obstetrics and Gynecology, Faculty of Medicine, The University of Tokyo, Tokyo 113-8655, Japan
| | - Yuichiro Miyamoto
- Department of Obstetrics and Gynecology, Faculty of Medicine, The University of Tokyo, Tokyo 113-8655, Japan
| | - Mayuyo Mori
- Department of Obstetrics and Gynecology, Faculty of Medicine, The University of Tokyo, Tokyo 113-8655, Japan
| | - Takayuki Iriyama
- Department of Obstetrics and Gynecology, Faculty of Medicine, The University of Tokyo, Tokyo 113-8655, Japan
| | - Osamu Wada-Hiraike
- Department of Obstetrics and Gynecology, Faculty of Medicine, The University of Tokyo, Tokyo 113-8655, Japan
| | - Yutaka Osuga
- Department of Obstetrics and Gynecology, Faculty of Medicine, The University of Tokyo, Tokyo 113-8655, Japan
| |
Collapse
|
22
|
Zang C, Turkcan MK, Narasimhan S, Cao Y, Yarali K, Xiang Z, Szot S, Ahmad F, Choksi S, Bitner DP, Filicori F, Kostic Z. Surgical Phase Recognition in Inguinal Hernia Repair-AI-Based Confirmatory Baseline and Exploration of Competitive Models. Bioengineering (Basel) 2023; 10:654. [PMID: 37370585 DOI: 10.3390/bioengineering10060654] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/05/2023] [Revised: 05/18/2023] [Accepted: 05/23/2023] [Indexed: 06/29/2023] Open
Abstract
Video-recorded robotic-assisted surgeries allow the use of automated computer vision and artificial intelligence/deep learning methods for quality assessment and workflow analysis in surgical phase recognition. We considered a dataset of 209 videos of robotic-assisted laparoscopic inguinal hernia repair (RALIHR) collected from 8 surgeons, defined rigorous ground-truth annotation rules, then pre-processed and annotated the videos. We deployed seven deep learning models to establish the baseline accuracy for surgical phase recognition and explored four advanced architectures. For rapid execution of the studies, we initially engaged three dozen MS-level engineering students in a competitive classroom setting, followed by focused research. We unified the data processing pipeline in a confirmatory study, and explored a number of scenarios which differ in how the DL networks were trained and evaluated. For the scenario with 21 validation videos of all surgeons, the Video Swin Transformer model achieved ~0.85 validation accuracy, and the Perceiver IO model achieved ~0.84. Our studies affirm the necessity of close collaborative research between medical experts and engineers for developing automated surgical phase recognition models deployable in clinical settings.
Collapse
Affiliation(s)
- Chengbo Zang
- Department of Electrical Engineering, Columbia University, New York, NY 10027, USA
| | - Mehmet Kerem Turkcan
- Department of Electrical Engineering, Columbia University, New York, NY 10027, USA
| | - Sanjeev Narasimhan
- Department of Computer Science, Columbia University, New York, NY 10027, USA
| | - Yuqing Cao
- Department of Electrical Engineering, Columbia University, New York, NY 10027, USA
| | - Kaan Yarali
- Department of Electrical Engineering, Columbia University, New York, NY 10027, USA
| | - Zixuan Xiang
- Department of Electrical Engineering, Columbia University, New York, NY 10027, USA
| | - Skyler Szot
- Department of Electrical Engineering, Columbia University, New York, NY 10027, USA
| | - Feroz Ahmad
- Department of Computer Science, Columbia University, New York, NY 10027, USA
| | - Sarah Choksi
- Intraoperative Performance Analytics Laboratory (IPAL), Lenox Hill Hospital, New York, NY 10021, USA
| | - Daniel P Bitner
- Intraoperative Performance Analytics Laboratory (IPAL), Lenox Hill Hospital, New York, NY 10021, USA
| | - Filippo Filicori
- Intraoperative Performance Analytics Laboratory (IPAL), Lenox Hill Hospital, New York, NY 10021, USA
- Zucker School of Medicine at Hofstra/Northwell Health, Hempstead, NY 11549, USA
| | - Zoran Kostic
- Department of Electrical Engineering, Columbia University, New York, NY 10027, USA
| |
Collapse
|
23
|
Nyangoh Timoh K, Huaulme A, Cleary K, Zaheer MA, Lavoué V, Donoho D, Jannin P. A systematic review of annotation for surgical process model analysis in minimally invasive surgery based on video. Surg Endosc 2023:10.1007/s00464-023-10041-w. [PMID: 37157035 DOI: 10.1007/s00464-023-10041-w] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/16/2022] [Accepted: 03/25/2023] [Indexed: 05/10/2023]
Abstract
BACKGROUND Annotated data are foundational to applications of supervised machine learning. However, there seems to be a lack of common language used in the field of surgical data science. The aim of this study is to review the process of annotation and semantics used in the creation of SPM for minimally invasive surgery videos. METHODS For this systematic review, we reviewed articles indexed in the MEDLINE database from January 2000 until March 2022. We selected articles using surgical video annotations to describe a surgical process model in the field of minimally invasive surgery. We excluded studies focusing on instrument detection or recognition of anatomical areas only. The risk of bias was evaluated with the Newcastle Ottawa Quality assessment tool. Data from the studies were visually presented in table using the SPIDER tool. RESULTS Of the 2806 articles identified, 34 were selected for review. Twenty-two were in the field of digestive surgery, six in ophthalmologic surgery only, one in neurosurgery, three in gynecologic surgery, and two in mixed fields. Thirty-one studies (88.2%) were dedicated to phase, step, or action recognition and mainly relied on a very simple formalization (29, 85.2%). Clinical information in the datasets was lacking for studies using available public datasets. The process of annotation for surgical process model was lacking and poorly described, and description of the surgical procedures was highly variable between studies. CONCLUSION Surgical video annotation lacks a rigorous and reproducible framework. This leads to difficulties in sharing videos between institutions and hospitals because of the different languages used. There is a need to develop and use common ontology to improve libraries of annotated surgical videos.
Collapse
Affiliation(s)
- Krystel Nyangoh Timoh
- Department of Gynecology and Obstetrics and Human Reproduction, CHU Rennes, Rennes, France.
- INSERM, LTSI - UMR 1099, University Rennes 1, Rennes, France.
- Laboratoire d'Anatomie et d'Organogenèse, Faculté de Médecine, Centre Hospitalier Universitaire de Rennes, 2 Avenue du Professeur Léon Bernard, 35043, Rennes Cedex, France.
- Department of Obstetrics and Gynecology, Rennes Hospital, Rennes, France.
| | - Arnaud Huaulme
- INSERM, LTSI - UMR 1099, University Rennes 1, Rennes, France
| | - Kevin Cleary
- Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Hospital, Washington, DC, 20010, USA
| | - Myra A Zaheer
- George Washington University School of Medicine and Health Sciences, Washington, DC, USA
| | - Vincent Lavoué
- Department of Gynecology and Obstetrics and Human Reproduction, CHU Rennes, Rennes, France
| | - Dan Donoho
- Division of Neurosurgery, Center for Neuroscience, Children's National Hospital, Washington, DC, 20010, USA
| | - Pierre Jannin
- INSERM, LTSI - UMR 1099, University Rennes 1, Rennes, France
| |
Collapse
|
24
|
Pan M, Wang S, Li J, Li J, Yang X, Liang K. An Automated Skill Assessment Framework Based on Visual Motion Signals and a Deep Neural Network in Robot-Assisted Minimally Invasive Surgery. SENSORS (BASEL, SWITZERLAND) 2023; 23:s23094496. [PMID: 37177699 PMCID: PMC10181496 DOI: 10.3390/s23094496] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/06/2023] [Revised: 04/27/2023] [Accepted: 05/03/2023] [Indexed: 05/15/2023]
Abstract
Surgical skill assessment can quantify the quality of the surgical operation via the motion state of the surgical instrument tip (SIT), which is considered one of the effective primary means by which to improve the accuracy of surgical operation. Traditional methods have displayed promising results in skill assessment. However, this success is predicated on the SIT sensors, making these approaches impractical when employing the minimally invasive surgical robot with such a tiny end size. To address the assessment issue regarding the operation quality of robot-assisted minimally invasive surgery (RAMIS), this paper proposes a new automatic framework for assessing surgical skills based on visual motion tracking and deep learning. The new method innovatively combines vision and kinematics. The kernel correlation filter (KCF) is introduced in order to obtain the key motion signals of the SIT and classify them by using the residual neural network (ResNet), realizing automated skill assessment in RAMIS. To verify its effectiveness and accuracy, the proposed method is applied to the public minimally invasive surgical robot dataset, the JIGSAWS. The results show that the method based on visual motion tracking technology and a deep neural network model can effectively and accurately assess the skill of robot-assisted surgery in near real-time. In a fairly short computational processing time of 3 to 5 s, the average accuracy of the assessment method is 92.04% and 84.80% in distinguishing two and three skill levels. This study makes an important contribution to the safe and high-quality development of RAMIS.
Collapse
Affiliation(s)
- Mingzhang Pan
- College of Mechanical Engineering, Guangxi University, Nanning 530004, China
- State Key Laboratory for Conservation and Utilization of Subtropical Agro-Bioresources, Nanning 530004, China
| | - Shuo Wang
- College of Mechanical Engineering, Guangxi University, Nanning 530004, China
| | - Jingao Li
- College of Mechanical Engineering, Guangxi University, Nanning 530004, China
| | - Jing Li
- College of Mechanical Engineering, Guangxi University, Nanning 530004, China
| | - Xiuze Yang
- College of Mechanical Engineering, Guangxi University, Nanning 530004, China
| | - Ke Liang
- College of Mechanical Engineering, Guangxi University, Nanning 530004, China
- Guangxi Key Laboratory of Manufacturing System & Advanced Manufacturing Technology, School of Mechanical Engineering, Guangxi University, Nanning 530004, China
| |
Collapse
|
25
|
Wu S, Chen Z, Liu R, Li A, Cao Y, Wei A, Liu Q, Liu J, Wang Y, Jiang J, Ying Z, An J, Peng B, Wang X. SurgSmart: an artificial intelligent system for quality control in laparoscopic cholecystectomy: an observational study. Int J Surg 2023; 109:1105-1114. [PMID: 37039533 PMCID: PMC10389595 DOI: 10.1097/js9.0000000000000329] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/29/2022] [Accepted: 02/22/2023] [Indexed: 04/12/2023]
Abstract
BACKGROUND The rate of bile duct injury in laparoscopic cholecystectomy (LC) continues to be high due to low critical view of safety (CVS) achievement and the absence of an effective quality control system. The development of an intelligent system enables the automatic quality control of LC surgery and, eventually, the mitigation of bile duct injury. This study aims to develop an intelligent surgical quality control system for LC and using the system to evaluate LC videos and investigate factors associated with CVS achievement. MATERIALS AND METHODS SurgSmart, an intelligent system capable of recognizing surgical phases, disease severity, critical division action, and CVS automatically, was developed using training datasets. SurgSmart was also applied in another multicenter dataset to validate its application and investigate factors associated with CVS achievement. RESULTS SurgSmart performed well in all models, with the critical division action model achieving the highest overall accuracy (98.49%), followed by the disease severity model (95.45%) and surgical phases model (88.61%). CVSI, CVSII, and CVSIII had an accuracy of 80.64, 97.62, and 78.87%, respectively. CVS was achieved in 4.33% in the system application dataset. In addition, the analysis indicated that surgeons at a higher hospital level had a higher CVS achievement rate. However, there was still considerable variation in CVS achievement among surgeons in the same hospital. CONCLUSIONS SurgSmart, the surgical quality control system, performed admirably in our study. In addition, the system's initial application demonstrated its broad potential for use in surgical quality control.
Collapse
Affiliation(s)
- Shangdi Wu
- Division of Pancreatic Surgery, Department of General Surgery
- West China School of Medicine
| | - Zixin Chen
- Division of Pancreatic Surgery, Department of General Surgery
- West China School of Medicine
| | - Runwen Liu
- ChengDu Withai Innovations Technology Company
| | - Ang Li
- Division of Pancreatic Surgery, Department of General Surgery
- Guang’an People’s Hospital, Guang’an, Sichuan Province, China
| | - Yu Cao
- Operating Room
- West China School of Nursing, Sichuan University
| | - Ailin Wei
- Guang’an People’s Hospital, Guang’an, Sichuan Province, China
| | | | - Jie Liu
- ChengDu Withai Innovations Technology Company
| | - Yuxian Wang
- ChengDu Withai Innovations Technology Company
| | - Jingwen Jiang
- West China Biomedical Big Data Center, West China Hospital of Sichuan University
- Med-X Center for Informatics, Sichuan University, Chengdu
| | - Zhiye Ying
- West China Biomedical Big Data Center, West China Hospital of Sichuan University
- Med-X Center for Informatics, Sichuan University, Chengdu
| | - Jingjing An
- Operating Room
- West China School of Nursing, Sichuan University
| | - Bing Peng
- Division of Pancreatic Surgery, Department of General Surgery
- West China School of Medicine
| | - Xin Wang
- Division of Pancreatic Surgery, Department of General Surgery
- West China School of Medicine
| |
Collapse
|
26
|
Chen KA, Kirchoff KE, Butler LR, Holloway AD, Kapadia MR, Gallagher KK, Gomez SM. Computer Vision Analysis of Specimen Mammography to Predict Margin Status. MEDRXIV : THE PREPRINT SERVER FOR HEALTH SCIENCES 2023:2023.03.06.23286864. [PMID: 36945565 PMCID: PMC10029028 DOI: 10.1101/2023.03.06.23286864] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 03/12/2023]
Abstract
Intra-operative specimen mammography is a valuable tool in breast cancer surgery, providing immediate assessment of margins for a resected tumor. However, the accuracy of specimen mammography in detecting microscopic margin positivity is low. We sought to develop a deep learning-based model to predict the pathologic margin status of resected breast tumors using specimen mammography. A dataset of specimen mammography images matched with pathology reports describing margin status was collected. Models pre-trained on radiologic images were developed and compared with models pre-trained on non-medical images. Model performance was assessed using sensitivity, specificity, and area under the receiver operating characteristic curve (AUROC). The dataset included 821 images and 53% had positive margins. For three out of four model architectures tested, models pre-trained on radiologic images outperformed domain-agnostic models. The highest performing model, InceptionV3, showed a sensitivity of 84%, a specificity of 42%, and AUROC of 0.71. These results compare favorably with the published literature on surgeon and radiologist interpretation of specimen mammography. With further development, these models could assist clinicians with identifying positive margins intra-operatively and decrease the rate of positive margins and re-operation in breast-conserving surgery.
Collapse
Affiliation(s)
- Kevin A Chen
- Department of Surgery, University of North Carolina at Chapel Hill, Chapel Hill, NC
| | - Kathryn E Kirchoff
- Joint Department of Biomedical Engineering, University of North Carolina at Chapel Hill, Chapel Hill, NC
| | - Logan R Butler
- Department of Surgery, University of North Carolina at Chapel Hill, Chapel Hill, NC
| | - Alexa D Holloway
- Department of Surgery, University of North Carolina at Chapel Hill, Chapel Hill, NC
| | - Muneera R Kapadia
- Department of Surgery, University of North Carolina at Chapel Hill, Chapel Hill, NC
| | | | - Shawn M Gomez
- Joint Department of Biomedical Engineering, University of North Carolina at Chapel Hill, Chapel Hill, NC
| |
Collapse
|
27
|
Capturing fine-grained details for video-based automation of suturing skills assessment. Int J Comput Assist Radiol Surg 2023; 18:545-552. [PMID: 36282465 PMCID: PMC9975072 DOI: 10.1007/s11548-022-02778-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/07/2022] [Accepted: 10/10/2022] [Indexed: 02/02/2023]
Abstract
OBJECTIVES Manually-collected suturing technical skill scores are strong predictors of continence recovery after robotic radical prostatectomy. Herein, we automate suturing technical skill scoring through computer vision (CV) methods as a scalable method to provide feedback. METHODS Twenty-two surgeons completed a suturing exercise three times on the Mimic™ Flex VR simulator. Instrument kinematic data (XYZ coordinates of each instrument and pose) were captured at 30 Hz. After standardized training, three human raters manually video segmented suturing task into four sub-stitch phases (Needle handling, Needle targeting, Needle driving, Needle withdrawal) and labeled the corresponding technical skill domains (Needle positioning, Needle entry, Needle driving, and Needle withdrawal). The CV framework extracted RGB features and optical flow frames using a pre-trained AlexNet. Additional CV strategies including auxiliary supervision (using kinematic data during training only) and attention mechanisms were implemented to improve performance. RESULTS This study included data from 15 expert surgeons (median caseload 300 [IQR 165-750]) and 7 training surgeons (0 [IQR 0-8]). In all, 226 virtual sutures were captured. Automated assessments for Needle positioning performed best with the simplest approach (1 s video; AUC 0.749). Remaining skill domains exhibited improvements with the implementation of auxiliary supervision and attention mechanisms when deployed separately (AUC 0.604-0.794). All techniques combined produced the best performance, particularly for Needle driving and Needle withdrawal (AUC 0.959 and 0.879, respectively). CONCLUSIONS This study demonstrated the best performance of automated suturing technical skills assessment to date using advanced CV techniques. Future work will determine if a "human in the loop" is necessary to verify surgeon evaluations.
Collapse
|
28
|
Cheikh Youssef S, Haram K, Noël J, Patel V, Porter J, Dasgupta P, Hachach-Haram N. Evolution of the digital operating room: the place of video technology in surgery. Langenbecks Arch Surg 2023; 408:95. [PMID: 36807211 PMCID: PMC9939374 DOI: 10.1007/s00423-023-02830-7] [Citation(s) in RCA: 7] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/25/2022] [Accepted: 02/06/2023] [Indexed: 02/23/2023]
Abstract
PURPOSE The aim of this review was to collate current evidence wherein digitalisation, through the incorporation of video technology and artificial intelligence (AI), is being applied to the practice of surgery. Applications are vast, and the literature investigating the utility of surgical video and its synergy with AI has steadily increased over the last 2 decades. This type of technology is widespread in other industries, such as autonomy in transportation and manufacturing. METHODS Articles were identified primarily using the PubMed and MEDLINE databases. The MeSH terms used were "surgical education", "surgical video", "video labelling", "surgery", "surgical workflow", "telementoring", "telemedicine", "machine learning", "deep learning" and "operating room". Given the breadth of the subject and the scarcity of high-level data in certain areas, a narrative synthesis was selected over a meta-analysis or systematic review to allow for a focussed discussion of the topic. RESULTS Three main themes were identified and analysed throughout this review, (1) the multifaceted utility of surgical video recording, (2) teleconferencing/telemedicine and (3) artificial intelligence in the operating room. CONCLUSIONS Evidence suggests the routine collection of intraoperative data will be beneficial in the advancement of surgery, by driving standardised, evidence-based surgical care and personalised training of future surgeons. However, many barriers stand in the way of widespread implementation, necessitating close collaboration between surgeons, data scientists, medicolegal personnel and hospital policy makers.
Collapse
Affiliation(s)
| | | | - Jonathan Noël
- Guy's and St. Thomas' NHS Foundation Trust, Urology Centre, King's Health Partners, London, UK
| | - Vipul Patel
- Adventhealth Global Robotics Institute, 400 Celebration Place, Celebration, FL, USA
| | - James Porter
- Department of Urology, Swedish Urology Group, Seattle, WA, USA
| | - Prokar Dasgupta
- Guy's and St. Thomas' NHS Foundation Trust, Urology Centre, King's Health Partners, London, UK
| | - Nadine Hachach-Haram
- Department of Plastic Surgery, Guy's and St. Thomas' NHS Foundation Trust, King's Health Partners, London, UK
| |
Collapse
|
29
|
Kulkarni CS, Deng S, Wang T, Hartman-Kenzler J, Barnes LE, Parker SH, Safford SD, Lau N. Scene-dependent, feedforward eye gaze metrics can differentiate technical skill levels of trainees in laparoscopic surgery. Surg Endosc 2023; 37:1569-1580. [PMID: 36123548 PMCID: PMC11062149 DOI: 10.1007/s00464-022-09582-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/07/2022] [Accepted: 08/25/2022] [Indexed: 10/14/2022]
Abstract
INTRODUCTION In laparoscopic surgery, looking in the target areas is an indicator of proficiency. However, gaze behaviors revealing feedforward control (i.e., looking ahead) and their importance have been under-investigated in surgery. This study aims to establish the sensitivity and relative importance of different scene-dependent gaze and motion metrics for estimating trainee proficiency levels in surgical skills. METHODS Medical students performed the Fundamentals of Laparoscopic Surgery peg transfer task while recording their gaze on the monitor and tool activities inside the trainer box. Using computer vision and fixation algorithms, five scene-dependent gaze metrics and one tool speed metric were computed for 499 practice trials. Cluster analysis on the six metrics was used to group the trials into different clusters/proficiency levels, and ANOVAs were conducted to test differences between proficiency levels. A Random Forest model was trained to study metric importance at predicting proficiency levels. RESULTS Three clusters were identified, corresponding to three proficiency levels. The correspondence between the clusters and proficiency levels was confirmed by differences between completion times (F2,488 = 38.94, p < .001). Further, ANOVAs revealed significant differences between the three levels for all six metrics. The Random Forest model predicted proficiency level with 99% out-of-bag accuracy and revealed that scene-dependent gaze metrics reflecting feedforward behaviors were more important for prediction than the ones reflecting feedback behaviors. CONCLUSION Scene-dependent gaze metrics revealed skill levels of trainees more precisely than between experts and novices as suggested in the literature. Further, feedforward gaze metrics appeared to be more important than feedback ones at predicting proficiency.
Collapse
Affiliation(s)
- Chaitanya S Kulkarni
- Grado Department of Industrial and Systems Engineering, Virginia Tech, 250 Durham Hall (0118), 1145 Perry Street, Blacksburg, VA, 24061, USA
| | - Shiyu Deng
- Grado Department of Industrial and Systems Engineering, Virginia Tech, 250 Durham Hall (0118), 1145 Perry Street, Blacksburg, VA, 24061, USA
| | - Tianzi Wang
- Grado Department of Industrial and Systems Engineering, Virginia Tech, 250 Durham Hall (0118), 1145 Perry Street, Blacksburg, VA, 24061, USA
| | | | - Laura E Barnes
- Environmental and Systems Engineering, University of Virginia, Charlottesville, VA, USA
| | | | - Shawn D Safford
- Division of Pediatric General and Thoracic Surgery, UPMC Children's Hospital of Pittsburgh, Harrisburg, PA, USA
| | - Nathan Lau
- Grado Department of Industrial and Systems Engineering, Virginia Tech, 250 Durham Hall (0118), 1145 Perry Street, Blacksburg, VA, 24061, USA.
| |
Collapse
|
30
|
Lai SL, Chen CS, Lin BR, Chang RF. Intraoperative Detection of Surgical Gauze Using Deep Convolutional Neural Network. Ann Biomed Eng 2023; 51:352-362. [PMID: 35972601 DOI: 10.1007/s10439-022-03033-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/05/2022] [Accepted: 07/19/2022] [Indexed: 01/25/2023]
Abstract
During laparoscopic surgery, surgical gauze is usually inserted into the body cavity to help achieve hemostasis. Retention of surgical gauze in the body cavity may necessitate reoperation and increase surgical risk. Using deep learning technology, this study aimed to propose a neural network model for gauze detection from the surgical video to record the presence of the gauze. The model was trained by the training group using YOLO (You Only Look Once)v5x6, then applied to the testing group. Positive predicted value (PPV), sensitivity, and mean average precision (mAP) were calculated. Furthermore, a timeline of gauze presence in the video was drawn by the model as well as human annotation to evaluate the accuracy. After the model was well-trained, the PPV, sensitivity, and mAP in the testing group were 0.920, 0.828, and 0.881, respectively. The inference time was 11.3 ms per image. The average accuracy of the model adding a marking and filtering process was 0.899. In conclusion, surgical gauze can be successfully detected using deep learning in the surgical video. Our model provided a fast detection of surgical gauze, allowing further real-time gauze tracing in laparoscopic surgery that may help surgeons recall the location of the missing gauze.
Collapse
Affiliation(s)
- Shuo-Lun Lai
- Graduate Institute of Biomedical Electronics and Bioinformatics, National Taiwan University, No.1, Sec.4, Roosevelt Road, Taipei, 10617, Taiwan.,Division of Colorectal Surgery, Department of Surgery, National Taiwan University Hospital and National Taiwan University College of Medicine, Taipei, Taiwan
| | - Chi-Sheng Chen
- Graduate Institute of Biomedical Electronics and Bioinformatics, National Taiwan University, No.1, Sec.4, Roosevelt Road, Taipei, 10617, Taiwan
| | - Been-Ren Lin
- Division of Colorectal Surgery, Department of Surgery, National Taiwan University Hospital and National Taiwan University College of Medicine, Taipei, Taiwan
| | - Ruey-Feng Chang
- Graduate Institute of Biomedical Electronics and Bioinformatics, National Taiwan University, No.1, Sec.4, Roosevelt Road, Taipei, 10617, Taiwan. .,Department of Computer Science and Information Engineering, National Taiwan University, Taipei, Taiwan.
| |
Collapse
|
31
|
Mansur A, Saleem Z, Elhakim T, Daye D. Role of artificial intelligence in risk prediction, prognostication, and therapy response assessment in colorectal cancer: current state and future directions. Front Oncol 2023; 13:1065402. [PMID: 36761957 PMCID: PMC9905815 DOI: 10.3389/fonc.2023.1065402] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/09/2022] [Accepted: 01/09/2023] [Indexed: 01/26/2023] Open
Abstract
Artificial Intelligence (AI) is a branch of computer science that utilizes optimization, probabilistic and statistical approaches to analyze and make predictions based on a vast amount of data. In recent years, AI has revolutionized the field of oncology and spearheaded novel approaches in the management of various cancers, including colorectal cancer (CRC). Notably, the applications of AI to diagnose, prognosticate, and predict response to therapy in CRC, is gaining traction and proving to be promising. There have also been several advancements in AI technologies to help predict metastases in CRC and in Computer-Aided Detection (CAD) Systems to improve miss rates for colorectal neoplasia. This article provides a comprehensive review of the role of AI in predicting risk, prognosis, and response to therapies among patients with CRC.
Collapse
Affiliation(s)
- Arian Mansur
- Harvard Medical School, Boston, MA, United States
| | | | - Tarig Elhakim
- Department of Radiology, Massachusetts General Hospital, Boston, MA, United States
| | - Dania Daye
- Department of Radiology, Massachusetts General Hospital, Boston, MA, United States,*Correspondence: Dania Daye,
| |
Collapse
|
32
|
Yin Z, Yao C, Zhang L, Qi S. Application of artificial intelligence in diagnosis and treatment of colorectal cancer: A novel Prospect. Front Med (Lausanne) 2023; 10:1128084. [PMID: 36968824 PMCID: PMC10030915 DOI: 10.3389/fmed.2023.1128084] [Citation(s) in RCA: 14] [Impact Index Per Article: 14.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/20/2022] [Accepted: 02/13/2023] [Indexed: 03/29/2023] Open
Abstract
In the past few decades, according to the rapid development of information technology, artificial intelligence (AI) has also made significant progress in the medical field. Colorectal cancer (CRC) is the third most diagnosed cancer worldwide, and its incidence and mortality rates are increasing yearly, especially in developing countries. This article reviews the latest progress in AI in diagnosing and treating CRC based on a systematic collection of previous literature. Most CRCs transform from polyp mutations. The computer-aided detection systems can significantly improve the polyp and adenoma detection rate by early colonoscopy screening, thereby lowering the possibility of mutating into CRC. Machine learning and bioinformatics analysis can help screen and identify more CRC biomarkers to provide the basis for non-invasive screening. The Convolutional neural networks can assist in reading histopathologic tissue images, reducing the experience difference among doctors. Various studies have shown that AI-based high-level auxiliary diagnostic systems can significantly improve the readability of medical images and help clinicians make more accurate diagnostic and therapeutic decisions. Moreover, Robotic surgery systems such as da Vinci have been more and more commonly used to treat CRC patients, according to their precise operating performance. The application of AI in neoadjuvant chemoradiotherapy has further improved the treatment and efficacy evaluation of CRC. In addition, AI represented by deep learning in gene sequencing research offers a new treatment option. All of these things have seen that AI has a promising prospect in the era of precision medicine.
Collapse
Affiliation(s)
- Zugang Yin
- Department of General Surgery, The First Affiliated Hospital of Dalian Medical University, Dalian, China
| | - Chenhui Yao
- Department of General Surgery, The First Affiliated Hospital of Dalian Medical University, Dalian, China
- *Correspondence: Chenhui Yao,
| | - Limin Zhang
- Department of Respiratory, The First Affiliated Hospital of Dalian Medical University, Dalian, China
| | - Shaohua Qi
- Institute of Laboratory Animal Science, Chinese Academy of Medical Sciences and Comparative Medicine Center, Peking Union Medical College, Beijing, China
| |
Collapse
|
33
|
Morris MX, Rajesh A, Asaad M, Hassan A, Saadoun R, Butler CE. Deep Learning Applications in Surgery: Current Uses and Future Directions. Am Surg 2023; 89:36-42. [PMID: 35567312 DOI: 10.1177/00031348221101490] [Citation(s) in RCA: 11] [Impact Index Per Article: 11.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/15/2022]
Abstract
Deep learning (DL) is a subset of machine learning that is rapidly gaining traction in surgical fields. Its tremendous capacity for powerful data-driven problem-solving has generated computational breakthroughs in many realms, with the fields of medicine and surgery becoming increasingly prominent avenues. Through its multi-layer architecture of interconnected neural networks, DL enables feature extraction and pattern recognition of highly complex and large-volume data. Across various surgical specialties, DL is being applied to optimize both preoperative planning and intraoperative performance in new and innovative ways. Surgeons are now able to integrate deep learning tools into their practice to improve patient safety and outcomes. Through this review, we explore the applications of deep learning in surgery and related subspecialties with an aim to shed light on the practical utilization of this technology in the present and near future.
Collapse
Affiliation(s)
- Miranda X Morris
- 12277Duke University School of Medicine, Durham, NC, USA.,101571Duke Pratt School of Engineering, Durham, NC, USA
| | - Aashish Rajesh
- Department of Surgery, 14742University of Texas Health Science Center at San Antonio, San Antonio, TX, USA
| | - Malke Asaad
- Department of Plastic Surgery, 6595University of Pittsburgh Medical Center, Pittsburgh, PA, USA
| | - Abbas Hassan
- Department of Plastic Surgery, 571198The University of Texas MD Anderson Cancer Center, Houston, TX, USA
| | - Rakan Saadoun
- Department of Plastic Surgery, 6595University of Pittsburgh Medical Center, Pittsburgh, PA, USA
| | - Charles E Butler
- Department of Plastic Surgery, 571198The University of Texas MD Anderson Cancer Center, Houston, TX, USA
| |
Collapse
|
34
|
Cheikh Youssef S, Hachach-Haram N, Aydin A, Shah TT, Sapre N, Nair R, Rai S, Dasgupta P. Video labelling robot-assisted radical prostatectomy and the role of artificial intelligence (AI): training a novice. J Robot Surg 2022; 17:695-701. [PMID: 36309954 PMCID: PMC9618152 DOI: 10.1007/s11701-022-01465-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/29/2022] [Accepted: 10/10/2022] [Indexed: 10/31/2022]
Abstract
AbstractVideo labelling is the assigning of meaningful information to raw videos. With the evolution of artificial intelligence and its intended incorporation into the operating room, video datasets can be invaluable tools for education and the training of intelligent surgical workflow systems through computer vision. However, the process of manual labelling of video datasets can prove costly and time-consuming for already busy practising surgeons. Twenty-five robot-assisted radical prostatectomy (RARP) procedures were recorded on Proximie, an augmented reality platform, anonymised and access given to a novice, who was trained to develop the knowledge and skills needed to accurately segment a full-length RARP procedure on a video labelling platform. A labelled video was subsequently randomly selected for assessment of accuracy by four practising urologists. Of the 25 videos allocated, 17 were deemed suitable for labelling, and 8 were excluded on the basis of procedure length and video quality. The labelled video selected for assessment was graded for accuracy of temporal labelling, with an average score of 93.1%, and a range of 85.6–100%. The self-training of a novice in the accurate segmentation of a surgical video to the standard of a practising urologist is feasible and practical for the RARP procedure. The assigning of temporal labels on a video labelling platform was also studied and proved feasible throughout the study period.
Collapse
|
35
|
Hartwig R, Berlet M, Czempiel T, Fuchtmann J, Rückert T, Feussner H, Wilhelm D. [Image-based supportive measures for future application in surgery]. CHIRURGIE (HEIDELBERG, GERMANY) 2022; 93:956-965. [PMID: 35737019 DOI: 10.1007/s00104-022-01668-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Accepted: 06/02/2022] [Indexed: 06/15/2023]
Abstract
BACKGROUND The development of assistive technologies will become of increasing importance in the coming years and not only in surgery. The comprehensive perception of the actual situation is the basis of every autonomous action. Different sensor systems can be used for this purpose, of which video-based systems have a special potential. METHOD Based on the available literature and on own research projects, central aspects of image-based support systems for surgery are presented. In this context, not only the potential but also the limitations of the methods are explained. RESULTS An established application is the phase detection of surgical interventions, for which surgical videos are analyzed using neural networks. Through a time-based and transformative analysis the results of the prediction could only recently be significantly improved. Robotic camera guidance systems will also use image data to autonomously navigate laparoscopes in the near future. The reliability of the systems needs to be adapted to the high requirements in surgery by means of additional information. A comparable multimodal approach has already been implemented for navigation and localization during laparoscopic procedures. For this purpose, video data are analyzed using various methods and these data are fused with other sensor modalities. DISCUSSION Image-based supportive methods are already available for various tasks and will become an important aspect for the surgery of the future; however, in order to be able to be reliably implemented for autonomous functions, they must be embedded in multimodal approaches in the future in order to provide the necessary security.
Collapse
Affiliation(s)
- R Hartwig
- Forschungsgruppe MITI, Klinik und Poliklinik für Chirurgie, Klinikum rechts der Isar, Technische Universität München, München, Deutschland
| | - M Berlet
- Forschungsgruppe MITI, Klinik und Poliklinik für Chirurgie, Klinikum rechts der Isar, Technische Universität München, München, Deutschland
- Fakultät für Medizin, Klinik und Poliklinik für Chirurgie, Klinikum rechts der Isar, Technische Universität München, München, Deutschland
| | - T Czempiel
- Computer Aided Medical Procedures, Technische Universitat München, München, Deutschland
| | - J Fuchtmann
- Forschungsgruppe MITI, Klinik und Poliklinik für Chirurgie, Klinikum rechts der Isar, Technische Universität München, München, Deutschland
| | - T Rückert
- Regensburg Medical Image Computing (ReMIC), Ostbayerische Technische Hochschule Regensburg (OTH Regensburg), Regensburg, Deutschland
| | - H Feussner
- Forschungsgruppe MITI, Klinik und Poliklinik für Chirurgie, Klinikum rechts der Isar, Technische Universität München, München, Deutschland
| | - D Wilhelm
- Forschungsgruppe MITI, Klinik und Poliklinik für Chirurgie, Klinikum rechts der Isar, Technische Universität München, München, Deutschland.
- Fakultät für Medizin, Klinik und Poliklinik für Chirurgie, Klinikum rechts der Isar, Technische Universität München, München, Deutschland.
| |
Collapse
|
36
|
Pangal DJ, Kugener G, Cardinal T, Lechtholz-Zey E, Collet C, Lasky S, Sundaram S, Zhu Y, Roshannai A, Chan J, Sinha A, Hung AJ, Anandkumar A, Zada G, Donoho DA. Use of surgical video-based automated performance metrics to predict blood loss and success of simulated vascular injury control in neurosurgery: a pilot study. J Neurosurg 2022; 137:840-849. [PMID: 34972086 DOI: 10.3171/2021.10.jns211064] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/29/2021] [Accepted: 10/06/2021] [Indexed: 11/06/2022]
Abstract
OBJECTIVE Experts can assess surgeon skill using surgical video, but a limited number of expert surgeons are available. Automated performance metrics (APMs) are a promising alternative but have not been created from operative videos in neurosurgery to date. The authors aimed to evaluate whether video-based APMs can predict task success and blood loss during endonasal endoscopic surgery in a validated cadaveric simulator of vascular injury of the internal carotid artery. METHODS Videos of cadaveric simulation trials by 73 neurosurgeons and otorhinolaryngologists were analyzed and manually annotated with bounding boxes to identify the surgical instruments in the frame. APMs in five domains were defined-instrument usage, time-to-phase, instrument disappearance, instrument movement, and instrument interactions-on the basis of expert analysis and task-specific surgical progressions. Bounding-box data of instrument position were then used to generate APMs for each trial. Multivariate linear regression was used to test for the associations between APMs and blood loss and task success (hemorrhage control in less than 5 minutes). The APMs of 93 successful trials were compared with the APMs of 49 unsuccessful trials. RESULTS In total, 29,151 frames of surgical video were annotated. Successful simulation trials had superior APMs in each domain, including proportionately more time spent with the key instruments in view (p < 0.001) and less time without hemorrhage control (p = 0.002). APMs in all domains improved in subsequent trials after the participants received personalized expert instruction. Attending surgeons had superior instrument usage, time-to-phase, and instrument disappearance metrics compared with resident surgeons (p < 0.01). APMs predicted surgeon performance better than surgeon training level or prior experience. A regression model that included APMs predicted blood loss with an R2 value of 0.87 (p < 0.001). CONCLUSIONS Video-based APMs were superior predictors of simulation trial success and blood loss than surgeon characteristics such as case volume and attending status. Surgeon educators can use APMs to assess competency, quantify performance, and provide actionable, structured feedback in order to improve patient outcomes. Validation of APMs provides a benchmark for further development of fully automated video assessment pipelines that utilize machine learning and computer vision.
Collapse
Affiliation(s)
- Dhiraj J Pangal
- 1Department of Neurosurgery, Keck School of Medicine of the University of Southern California, Los Angeles, California
| | - Guillaume Kugener
- 1Department of Neurosurgery, Keck School of Medicine of the University of Southern California, Los Angeles, California
| | - Tyler Cardinal
- 1Department of Neurosurgery, Keck School of Medicine of the University of Southern California, Los Angeles, California
| | - Elizabeth Lechtholz-Zey
- 1Department of Neurosurgery, Keck School of Medicine of the University of Southern California, Los Angeles, California
| | - Casey Collet
- 1Department of Neurosurgery, Keck School of Medicine of the University of Southern California, Los Angeles, California
| | - Sasha Lasky
- 1Department of Neurosurgery, Keck School of Medicine of the University of Southern California, Los Angeles, California
| | - Shivani Sundaram
- 1Department of Neurosurgery, Keck School of Medicine of the University of Southern California, Los Angeles, California
| | - Yichao Zhu
- 1Department of Neurosurgery, Keck School of Medicine of the University of Southern California, Los Angeles, California
| | - Arman Roshannai
- 1Department of Neurosurgery, Keck School of Medicine of the University of Southern California, Los Angeles, California
| | - Justin Chan
- 1Department of Neurosurgery, Keck School of Medicine of the University of Southern California, Los Angeles, California
| | - Aditya Sinha
- 1Department of Neurosurgery, Keck School of Medicine of the University of Southern California, Los Angeles, California
| | - Andrew J Hung
- 2Center for Robotic Simulation and Education, USC Institute of Urology, Keck School of Medicine of the University of Southern California, Los Angeles, California
| | - Animashree Anandkumar
- 3Computing + Mathematical Sciences, California Institute of Technology, Pasadena, California; and
| | - Gabriel Zada
- 1Department of Neurosurgery, Keck School of Medicine of the University of Southern California, Los Angeles, California
| | - Daniel A Donoho
- 4Division of Neurosurgery, Center for Neuroscience, Children's National Medical Center, Washington, DC
| |
Collapse
|
37
|
Sasaki K, Ito M, Kobayashi S, Kitaguchi D, Matsuzaki H, Kudo M, Hasegawa H, Takeshita N, Sugimoto M, Mitsunaga S, Gotohda N. Automated surgical workflow identification by artificial intelligence in laparoscopic hepatectomy: Experimental research. Int J Surg 2022; 105:106856. [PMID: 36031068 DOI: 10.1016/j.ijsu.2022.106856] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/21/2022] [Accepted: 08/19/2022] [Indexed: 12/15/2022]
Abstract
BACKGROUND To perform accurate laparoscopic hepatectomy (LH) without injury, novel intraoperative systems of computer-assisted surgery (CAS) for LH are expected. Automated surgical workflow identification is a key component for developing CAS systems. This study aimed to develop a deep-learning model for automated surgical step identification in LH. MATERIALS AND METHODS We constructed a dataset comprising 40 cases of pure LH videos; 30 and 10 cases were used for the training and testing datasets, respectively. Each video was divided into 30 frames per second as static images. LH was divided into nine surgical steps (Steps 0-8), and each frame was annotated as being within one of these steps in the training set. After extracorporeal actions (Step 0) were excluded from the video, two deep-learning models of automated surgical step identification for 8-step and 6-step models were developed using a convolutional neural network (Models 1 & 2). Each frame in the testing dataset was classified using the constructed model performed in real-time. RESULTS Above 8 million frames were annotated for surgical step identification from the pure LH videos. The overall accuracy of Model 1 was 0.891, which was increased to 0.947 in Model 2. Median and average accuracy for each case in Model 2 was 0.927 (range, 0.884-0.997) and 0.937 ± 0.04 (standardized difference), respectively. Real-time automated surgical step identification was performed at 21 frames per second. CONCLUSIONS We developed a highly accurate deep-learning model for surgical step identification in pure LH. Our model could be applied to intraoperative systems of CAS.
Collapse
Affiliation(s)
- Kimimasa Sasaki
- Surgical Device Innovation Office, National Cancer Center Hospital East, 6-5-1, Kashiwanoha, Kashiwa-City, Chiba, 277-8577, Japan; Department of Hepatobiliary and Pancreatic Surgery, National Cancer Center Hospital East, 6-5-1, Kashiwanoha, Kashiwa-City, Chiba, 277-8577, Japan; Course of Advanced Clinical Research of Cancer, Juntendo University Graduate School of Medicine, 2-1-1, Hongo, Bunkyo-Ward, Tokyo, 113-8421, Japan
| | - Masaaki Ito
- Surgical Device Innovation Office, National Cancer Center Hospital East, 6-5-1, Kashiwanoha, Kashiwa-City, Chiba, 277-8577, Japan.
| | - Shin Kobayashi
- Department of Hepatobiliary and Pancreatic Surgery, National Cancer Center Hospital East, 6-5-1, Kashiwanoha, Kashiwa-City, Chiba, 277-8577, Japan
| | - Daichi Kitaguchi
- Surgical Device Innovation Office, National Cancer Center Hospital East, 6-5-1, Kashiwanoha, Kashiwa-City, Chiba, 277-8577, Japan
| | - Hiroki Matsuzaki
- Surgical Device Innovation Office, National Cancer Center Hospital East, 6-5-1, Kashiwanoha, Kashiwa-City, Chiba, 277-8577, Japan
| | - Masashi Kudo
- Department of Hepatobiliary and Pancreatic Surgery, National Cancer Center Hospital East, 6-5-1, Kashiwanoha, Kashiwa-City, Chiba, 277-8577, Japan
| | - Hiro Hasegawa
- Surgical Device Innovation Office, National Cancer Center Hospital East, 6-5-1, Kashiwanoha, Kashiwa-City, Chiba, 277-8577, Japan
| | - Nobuyoshi Takeshita
- Surgical Device Innovation Office, National Cancer Center Hospital East, 6-5-1, Kashiwanoha, Kashiwa-City, Chiba, 277-8577, Japan
| | - Motokazu Sugimoto
- Department of Hepatobiliary and Pancreatic Surgery, National Cancer Center Hospital East, 6-5-1, Kashiwanoha, Kashiwa-City, Chiba, 277-8577, Japan
| | - Shuichi Mitsunaga
- Course of Advanced Clinical Research of Cancer, Juntendo University Graduate School of Medicine, 2-1-1, Hongo, Bunkyo-Ward, Tokyo, 113-8421, Japan; Department of Hepatobiliary and Pancreatic Oncology, National Cancer Center Hospital East, 6-5-1, Kashiwanoha, Kashiwa-City, Chiba, 277-8577, Japan
| | - Naoto Gotohda
- Department of Hepatobiliary and Pancreatic Surgery, National Cancer Center Hospital East, 6-5-1, Kashiwanoha, Kashiwa-City, Chiba, 277-8577, Japan; Course of Advanced Clinical Research of Cancer, Juntendo University Graduate School of Medicine, 2-1-1, Hongo, Bunkyo-Ward, Tokyo, 113-8421, Japan
| |
Collapse
|
38
|
Multi-Camera Multi-Person Tracking and Re-Identification in an Operating Room. J Imaging 2022; 8:jimaging8080219. [PMID: 36005462 PMCID: PMC9410347 DOI: 10.3390/jimaging8080219] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/02/2022] [Revised: 08/08/2022] [Accepted: 08/14/2022] [Indexed: 11/20/2022] Open
Abstract
Multi-camera multi-person (MCMP) tracking and re-identification (ReID) are essential tasks in safety, pedestrian analysis, and so on; however, most research focuses on outdoor scenarios because they are much more complicated to deal with occlusions and misidentification in a crowded room with obstacles. Moreover, it is challenging to complete the two tasks in one framework. We present a trajectory-based method, integrating tracking and ReID tasks. First, the poses of all surgical members captured by each camera are detected frame-by-frame; then, the detected poses are exploited to track the trajectories of all members for each camera; finally, these trajectories of different cameras are clustered to re-identify the members in the operating room across all cameras. Compared to other MCMP tracking and ReID methods, the proposed one mainly exploits trajectories, taking texture features that are less distinguishable in the operating room scenario as auxiliary cues. We also integrate temporal information during ReID, which is more reliable than the state-of-the-art framework where ReID is conducted frame-by-frame. In addition, our framework requires no training before deployment in new scenarios. We also created an annotated MCMP dataset with actual operating room videos. Our experiments prove the effectiveness of the proposed trajectory-based ReID algorithm. The proposed framework achieves 85.44% accuracy in the ReID task, outperforming the state-of-the-art framework in our operating room dataset.
Collapse
|
39
|
Quero G, Mascagni P, Kolbinger FR, Fiorillo C, De Sio D, Longo F, Schena CA, Laterza V, Rosa F, Menghi R, Papa V, Tondolo V, Cina C, Distler M, Weitz J, Speidel S, Padoy N, Alfieri S. Artificial Intelligence in Colorectal Cancer Surgery: Present and Future Perspectives. Cancers (Basel) 2022; 14:cancers14153803. [PMID: 35954466 PMCID: PMC9367568 DOI: 10.3390/cancers14153803] [Citation(s) in RCA: 9] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/28/2022] [Revised: 07/29/2022] [Accepted: 08/03/2022] [Indexed: 02/05/2023] Open
Abstract
Artificial intelligence (AI) and computer vision (CV) are beginning to impact medicine. While evidence on the clinical value of AI-based solutions for the screening and staging of colorectal cancer (CRC) is mounting, CV and AI applications to enhance the surgical treatment of CRC are still in their early stage. This manuscript introduces key AI concepts to a surgical audience, illustrates fundamental steps to develop CV for surgical applications, and provides a comprehensive overview on the state-of-the-art of AI applications for the treatment of CRC. Notably, studies show that AI can be trained to automatically recognize surgical phases and actions with high accuracy even in complex colorectal procedures such as transanal total mesorectal excision (TaTME). In addition, AI models were trained to interpret fluorescent signals and recognize correct dissection planes during total mesorectal excision (TME), suggesting CV as a potentially valuable tool for intraoperative decision-making and guidance. Finally, AI could have a role in surgical training, providing automatic surgical skills assessment in the operating room. While promising, these proofs of concept require further development, validation in multi-institutional data, and clinical studies to confirm AI as a valuable tool to enhance CRC treatment.
Collapse
Affiliation(s)
- Giuseppe Quero
- Digestive Surgery Unit, Fondazione Policlinico Universitario A. Gemelli IRCCS, Largo Agostino Gemelli 8, 00168 Rome, Italy
- Faculty of Medicine, Università Cattolica del Sacro Cuore di Roma, Largo Francesco Vito 1, 00168 Rome, Italy
| | - Pietro Mascagni
- Faculty of Medicine, Università Cattolica del Sacro Cuore di Roma, Largo Francesco Vito 1, 00168 Rome, Italy
- Institute of Image-Guided Surgery, IHU-Strasbourg, 67000 Strasbourg, France
| | - Fiona R. Kolbinger
- Department for Visceral, Thoracic and Vascular Surgery, University Hospital and Faculty of Medicine Carl Gustav Carus, Technische Universität Dresden, 01307 Dresden, Germany
| | - Claudio Fiorillo
- Digestive Surgery Unit, Fondazione Policlinico Universitario A. Gemelli IRCCS, Largo Agostino Gemelli 8, 00168 Rome, Italy
- Correspondence: ; Tel.: +39-333-8747996
| | - Davide De Sio
- Digestive Surgery Unit, Fondazione Policlinico Universitario A. Gemelli IRCCS, Largo Agostino Gemelli 8, 00168 Rome, Italy
| | - Fabio Longo
- Digestive Surgery Unit, Fondazione Policlinico Universitario A. Gemelli IRCCS, Largo Agostino Gemelli 8, 00168 Rome, Italy
| | - Carlo Alberto Schena
- Digestive Surgery Unit, Fondazione Policlinico Universitario A. Gemelli IRCCS, Largo Agostino Gemelli 8, 00168 Rome, Italy
- Faculty of Medicine, Università Cattolica del Sacro Cuore di Roma, Largo Francesco Vito 1, 00168 Rome, Italy
| | - Vito Laterza
- Digestive Surgery Unit, Fondazione Policlinico Universitario A. Gemelli IRCCS, Largo Agostino Gemelli 8, 00168 Rome, Italy
- Faculty of Medicine, Università Cattolica del Sacro Cuore di Roma, Largo Francesco Vito 1, 00168 Rome, Italy
| | - Fausto Rosa
- Digestive Surgery Unit, Fondazione Policlinico Universitario A. Gemelli IRCCS, Largo Agostino Gemelli 8, 00168 Rome, Italy
- Faculty of Medicine, Università Cattolica del Sacro Cuore di Roma, Largo Francesco Vito 1, 00168 Rome, Italy
| | - Roberta Menghi
- Digestive Surgery Unit, Fondazione Policlinico Universitario A. Gemelli IRCCS, Largo Agostino Gemelli 8, 00168 Rome, Italy
- Faculty of Medicine, Università Cattolica del Sacro Cuore di Roma, Largo Francesco Vito 1, 00168 Rome, Italy
| | - Valerio Papa
- Digestive Surgery Unit, Fondazione Policlinico Universitario A. Gemelli IRCCS, Largo Agostino Gemelli 8, 00168 Rome, Italy
- Faculty of Medicine, Università Cattolica del Sacro Cuore di Roma, Largo Francesco Vito 1, 00168 Rome, Italy
| | - Vincenzo Tondolo
- Digestive Surgery Unit, Fondazione Policlinico Universitario A. Gemelli IRCCS, Largo Agostino Gemelli 8, 00168 Rome, Italy
| | - Caterina Cina
- Digestive Surgery Unit, Fondazione Policlinico Universitario A. Gemelli IRCCS, Largo Agostino Gemelli 8, 00168 Rome, Italy
| | - Marius Distler
- Department for Visceral, Thoracic and Vascular Surgery, University Hospital and Faculty of Medicine Carl Gustav Carus, Technische Universität Dresden, 01307 Dresden, Germany
| | - Juergen Weitz
- Department for Visceral, Thoracic and Vascular Surgery, University Hospital and Faculty of Medicine Carl Gustav Carus, Technische Universität Dresden, 01307 Dresden, Germany
| | - Stefanie Speidel
- National Center for Tumor Diseases (NCT), Partner Site Dresden, 01307 Dresden, Germany
| | - Nicolas Padoy
- Institute of Image-Guided Surgery, IHU-Strasbourg, 67000 Strasbourg, France
- ICube, Centre National de la Recherche Scientifique (CNRS), University of Strasbourg, 67000 Strasbourg, France
| | - Sergio Alfieri
- Digestive Surgery Unit, Fondazione Policlinico Universitario A. Gemelli IRCCS, Largo Agostino Gemelli 8, 00168 Rome, Italy
- Faculty of Medicine, Università Cattolica del Sacro Cuore di Roma, Largo Francesco Vito 1, 00168 Rome, Italy
| |
Collapse
|
40
|
Gumbs AA, Grasso V, Bourdel N, Croner R, Spolverato G, Frigerio I, Illanes A, Abu Hilal M, Park A, Elyan E. The Advances in Computer Vision That Are Enabling More Autonomous Actions in Surgery: A Systematic Review of the Literature. SENSORS 2022; 22:s22134918. [PMID: 35808408 PMCID: PMC9269548 DOI: 10.3390/s22134918] [Citation(s) in RCA: 9] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 05/20/2022] [Revised: 06/21/2022] [Accepted: 06/21/2022] [Indexed: 12/28/2022]
Abstract
This is a review focused on advances and current limitations of computer vision (CV) and how CV can help us obtain to more autonomous actions in surgery. It is a follow-up article to one that we previously published in Sensors entitled, “Artificial Intelligence Surgery: How Do We Get to Autonomous Actions in Surgery?” As opposed to that article that also discussed issues of machine learning, deep learning and natural language processing, this review will delve deeper into the field of CV. Additionally, non-visual forms of data that can aid computerized robots in the performance of more autonomous actions, such as instrument priors and audio haptics, will also be highlighted. Furthermore, the current existential crisis for surgeons, endoscopists and interventional radiologists regarding more autonomy during procedures will be discussed. In summary, this paper will discuss how to harness the power of CV to keep doctors who do interventions in the loop.
Collapse
Affiliation(s)
- Andrew A. Gumbs
- Departement de Chirurgie Digestive, Centre Hospitalier Intercommunal de, Poissy/Saint-Germain-en-Laye, 78300 Poissy, France
- Department of Surgery, University of Magdeburg, 39106 Magdeburg, Germany;
- Correspondence: ; Tel.: +33-139274873
| | - Vincent Grasso
- Family Christian Health Center, 31 West 155th St., Harvey, IL 60426, USA;
| | - Nicolas Bourdel
- Gynecological Surgery Department, CHU Clermont Ferrand, 1, Place Lucie-Aubrac Clermont-Ferrand, 63100 Clermont-Ferrand, France;
- EnCoV, Institut Pascal, UMR6602 CNRS, UCA, Clermont-Ferrand University Hospital, 63000 Clermont-Ferrand, France
- SurgAR-Surgical Augmented Reality, 63000 Clermont-Ferrand, France
| | - Roland Croner
- Department of Surgery, University of Magdeburg, 39106 Magdeburg, Germany;
| | - Gaya Spolverato
- Department of Surgical, Oncological and Gastroenterological Sciences, University of Padova, 35122 Padova, Italy;
| | - Isabella Frigerio
- Department of Hepato-Pancreato-Biliary Surgery, Pederzoli Hospital, 37019 Peschiera del Garda, Italy;
| | - Alfredo Illanes
- INKA-Innovation Laboratory for Image Guided Therapy, Otto-von-Guericke University Magdeburg, 39120 Magdeburg, Germany;
| | - Mohammad Abu Hilal
- Unità Chirurgia Epatobiliopancreatica, Robotica e Mininvasiva, Fondazione Poliambulanza Istituto Ospedaliero, Via Bissolati, 57, 25124 Brescia, Italy;
| | - Adrian Park
- Anne Arundel Medical Center, Johns Hopkins University, Annapolis, MD 21401, USA;
| | - Eyad Elyan
- School of Computing, Robert Gordon University, Aberdeen AB10 7JG, UK;
| |
Collapse
|
41
|
Tousignant MR, Liu X, Ershad Langroodi M, Jarc AM. Identification of Main Influencers of Surgical Efficiency and Variability Using Task-Level Objective Metrics: A Five-Year Robotic Sleeve Gastrectomy Case Series. Front Surg 2022; 9:756522. [PMID: 35586509 PMCID: PMC9108208 DOI: 10.3389/fsurg.2022.756522] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/10/2021] [Accepted: 03/07/2022] [Indexed: 11/13/2022] Open
Abstract
Objective Surgical efficiency and variability are critical contributors to optimal outcomes, patient experience, care team experience, and total cost to treat per disease episode. Opportunities remain to develop scalable, objective methods to quantify surgical behaviors that maximize efficiency and reduce variability. Such objective measures can then be used to provide surgeons with timely and user-specific feedbacks to monitor performances and facilitate training and learning. In this study, we used objective task-level analysis to identify dominant contributors toward surgical efficiency and variability across the procedural steps of robotic-assisted sleeve gastrectomy (RSG) over a five-year period for a single surgeon. These results enable actionable insights that can both complement those from population level analyses and be tailored to an individual surgeon's practice and experience. Methods Intraoperative video recordings of 77 RSG procedures performed by a single surgeon from 2015 to 2019 were reviewed and segmented into surgical tasks. Surgeon-initiated events when controlling the robotic-assisted surgical system were used to compute objective metrics. A series of multi-staged regression analysis were used to determine: if any specific tasks or patient body mass index (BMI) statistically impacted procedure duration; which objective metrics impacted critical task efficiency; and which task(s) statistically contributed to procedure variability. Results Stomach dissection was found to be the most significant contributor to procedure duration (β = 0.344, p< 0.001; R = 0.81, p< 0.001) followed by surgical inactivity and stomach stapling. Patient BMI was not found to be statistically significantly correlated with procedure duration (R = -0.01, p = 0.90). Energy activation rate, a robotic system event-based metric, was identified as a dominant feature in predicting stomach dissection duration and differentiating earlier and later case groups. Reduction of procedure variability was observed between earlier (2015-2016) and later (2017-2019) groups (IQR = 14.20 min vs. 6.79 min). Stomach dissection was found to contribute most to procedure variability (β = 0.74, p < 0.001). Conclusions A surgical task-based objective analysis was used to identify major contributors to surgical efficiency and variability. We believe this data-driven method will enable clinical teams to quantify surgeon-specific performance and identify actionable opportunities focused on the dominant surgical tasks impacting overall procedure efficiency and consistency.
Collapse
Affiliation(s)
- Mark R. Tousignant
- Medical Safety and Innovation, Intuitive Surgical Inc., Sunnyvale, CA, United States
| | - Xi Liu
- Applied Research, Intuitive Surgical Inc., Peachtree Corners, GA, United States
| | | | - Anthony M. Jarc
- Applied Research, Intuitive Surgical Inc., Peachtree Corners, GA, United States
| |
Collapse
|
42
|
Shinozuka K, Turuda S, Fujinaga A, Nakanuma H, Kawamura M, Matsunobu Y, Tanaka Y, Kamiyama T, Ebe K, Endo Y, Etoh T, Inomata M, Tokuyasu T. Artificial intelligence software available for medical devices: surgical phase recognition in laparoscopic cholecystectomy. Surg Endosc 2022; 36:7444-7452. [PMID: 35266049 PMCID: PMC9485170 DOI: 10.1007/s00464-022-09160-7] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/18/2021] [Accepted: 02/18/2022] [Indexed: 11/29/2022]
Abstract
Background Surgical process modeling automatically identifies surgical phases, and further improvement in recognition accuracy is expected with deep learning. Surgical tool or time series information has been used to improve the recognition accuracy of a model. However, it is difficult to collect this information continuously intraoperatively. The present study aimed to develop a deep convolution neural network (CNN) model that correctly identifies the surgical phase during laparoscopic cholecystectomy (LC). Methods We divided LC into six surgical phases (P1–P6) and one redundant phase (P0). We prepared 115 LC videos and converted them to image frames at 3 fps. Three experienced doctors labeled the surgical phases in all image frames. Our deep CNN model was trained with 106 of the 115 annotation datasets and was evaluated with the remaining datasets. By depending on both the prediction probability and frequency for a certain period, we aimed for highly accurate surgical phase recognition in the operation room. Results Nine full LC videos were converted into image frames and were fed to our deep CNN model. The average accuracy, precision, and recall were 0.970, 0.855, and 0.863, respectively. Conclusion The deep learning CNN model in this study successfully identified both the six surgical phases and the redundant phase, P0, which may increase the versatility of the surgical process recognition model for clinical use. We believe that this model can be used in artificial intelligence for medical devices. The degree of recognition accuracy is expected to improve with developments in advanced deep learning algorithms.
Collapse
Affiliation(s)
- Ken'ichi Shinozuka
- Faculty of Information Engineering, Department of Information and Systems Engineering, Fukuoka Institute of Technology, 1-30-1 Wajiro higashi, Higashi-ku, Fukuoka, Fukuoka, 811-0295, Japan
| | - Sayaka Turuda
- Faculty of Information Engineering, Department of Information and Systems Engineering, Fukuoka Institute of Technology, 1-30-1 Wajiro higashi, Higashi-ku, Fukuoka, Fukuoka, 811-0295, Japan
| | - Atsuro Fujinaga
- Faculty of Medicine, Department of Gastroenterological and Pediatric Surgery, Oita University, Oita, Japan
| | - Hiroaki Nakanuma
- Faculty of Medicine, Department of Gastroenterological and Pediatric Surgery, Oita University, Oita, Japan
| | - Masahiro Kawamura
- Faculty of Medicine, Department of Gastroenterological and Pediatric Surgery, Oita University, Oita, Japan
| | - Yusuke Matsunobu
- Faculty of Information Engineering, Department of Information and Systems Engineering, Fukuoka Institute of Technology, 1-30-1 Wajiro higashi, Higashi-ku, Fukuoka, Fukuoka, 811-0295, Japan
| | - Yuki Tanaka
- Customer Solutions Development, Platform Technology, Olympus Technologies Asia, Olympus Corporation, Tokyo, Japan
| | - Toshiya Kamiyama
- Customer Solutions Development, Platform Technology, Olympus Technologies Asia, Olympus Corporation, Tokyo, Japan
| | - Kohei Ebe
- Customer Solutions Development, Platform Technology, Olympus Technologies Asia, Olympus Corporation, Tokyo, Japan
| | - Yuichi Endo
- Faculty of Medicine, Department of Gastroenterological and Pediatric Surgery, Oita University, Oita, Japan
| | - Tsuyoshi Etoh
- Faculty of Medicine, Department of Gastroenterological and Pediatric Surgery, Oita University, Oita, Japan
| | - Masafumi Inomata
- Faculty of Medicine, Department of Gastroenterological and Pediatric Surgery, Oita University, Oita, Japan
| | - Tatsushi Tokuyasu
- Faculty of Information Engineering, Department of Information and Systems Engineering, Fukuoka Institute of Technology, 1-30-1 Wajiro higashi, Higashi-ku, Fukuoka, Fukuoka, 811-0295, Japan.
| |
Collapse
|
43
|
Kugener G, Pangal DJ, Cardinal T, Collet C, Lechtholz-Zey E, Lasky S, Sundaram S, Markarian N, Zhu Y, Roshannai A, Sinha A, Han XY, Papyan V, Hung A, Anandkumar A, Wrobel B, Zada G, Donoho DA. Utility of the Simulated Outcomes Following Carotid Artery Laceration Video Data Set for Machine Learning Applications. JAMA Netw Open 2022; 5:e223177. [PMID: 35311962 PMCID: PMC8938712 DOI: 10.1001/jamanetworkopen.2022.3177] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 01/28/2023] Open
Abstract
IMPORTANCE Surgical data scientists lack video data sets that depict adverse events, which may affect model generalizability and introduce bias. Hemorrhage may be particularly challenging for computer vision-based models because blood obscures the scene. OBJECTIVE To assess the utility of the Simulated Outcomes Following Carotid Artery Laceration (SOCAL)-a publicly available surgical video data set of hemorrhage complication management with instrument annotations and task outcomes-to provide benchmarks for surgical data science techniques, including computer vision instrument detection, instrument use metrics and outcome associations, and validation of a SOCAL-trained neural network using real operative video. DESIGN, SETTING, AND PARTICIPANTS For this quailty improvement study, a total of 75 surgeons with 1 to 30 years' experience (mean, 7 years) were filmed from January 1, 2017, to December 31, 2020, managing catastrophic surgical hemorrhage in a high-fidelity cadaveric training exercise at nationwide training courses. Videos were annotated from January 1 to June 30, 2021. INTERVENTIONS Surgeons received expert coaching between 2 trials. MAIN OUTCOMES AND MEASURES Hemostasis within 5 minutes (task success, dichotomous), time to hemostasis (in seconds), and blood loss (in milliliters) were recorded. Deep neural networks (DNNs) were trained to detect surgical instruments in view. Model performance was measured using mean average precision (mAP), sensitivity, and positive predictive value. RESULTS SOCAL contains 31 443 frames with 65 071 surgical instrument annotations from 147 trials with associated surgeon demographic characteristics, time to hemostasis, and recorded blood loss for each trial. Computer vision-based instrument detection methods using DNNs trained on SOCAL achieved a mAP of 0.67 overall and 0.91 for the most common surgical instrument (suction). Hemorrhage control challenges standard object detectors: detection of some surgical instruments remained poor (mAP, 0.25). On real intraoperative video, the model achieved a sensitivity of 0.77 and a positive predictive value of 0.96. Instrument use metrics derived from the SOCAL video were significantly associated with performance (blood loss). CONCLUSIONS AND RELEVANCE Hemorrhage control is a high-stakes adverse event that poses unique challenges for video analysis, but no data sets of hemorrhage control exist. The use of SOCAL, the first data set to depict hemorrhage control, allows the benchmarking of data science applications, including object detection, performance metric development, and identification of metrics associated with outcomes. In the future, SOCAL may be used to build and validate surgical data science models.
Collapse
Affiliation(s)
- Guillaume Kugener
- Department of Neurosurgery, Keck School of Medicine of the University of Southern California, Los Angeles
| | - Dhiraj J. Pangal
- Department of Neurosurgery, Keck School of Medicine of the University of Southern California, Los Angeles
| | - Tyler Cardinal
- Department of Neurosurgery, Keck School of Medicine of the University of Southern California, Los Angeles
| | - Casey Collet
- Department of Neurosurgery, Keck School of Medicine of the University of Southern California, Los Angeles
| | - Elizabeth Lechtholz-Zey
- Department of Neurosurgery, Keck School of Medicine of the University of Southern California, Los Angeles
| | - Sasha Lasky
- Department of Neurosurgery, Keck School of Medicine of the University of Southern California, Los Angeles
| | - Shivani Sundaram
- Department of Neurosurgery, Keck School of Medicine of the University of Southern California, Los Angeles
| | - Nicholas Markarian
- Department of Neurosurgery, Keck School of Medicine of the University of Southern California, Los Angeles
| | - Yichao Zhu
- Department of Computer Science, Viterbi School of Engineering, University of Southern California, Los Angeles
| | - Arman Roshannai
- Department of Neurosurgery, Keck School of Medicine of the University of Southern California, Los Angeles
| | - Aditya Sinha
- Department of Neurosurgery, Keck School of Medicine of the University of Southern California, Los Angeles
| | - X. Y. Han
- Department of Operations Research and Information Engineering, Cornell University, Ithaca, New York
| | - Vardan Papyan
- Department of Mathematics, University of Toronto, Toronto, Ontario, Canada
| | - Andrew Hung
- Center for Robotic Simulation and Education, USC Institute of Urology, Keck School of Medicine of the University of Southern California, Los Angeles
| | - Animashree Anandkumar
- Department of Computer Science and Mathematics, California Institute of Technology, Pasadena
| | - Bozena Wrobel
- Department of Otolaryngology, Keck School of Medicine of the University of Southern California, Los Angeles
| | - Gabriel Zada
- Department of Neurosurgery, Keck School of Medicine of the University of Southern California, Los Angeles
| | - Daniel A. Donoho
- Division of Neurosurgery, Center for Neuroscience, Children’s National Hospital, Washington, DC
| |
Collapse
|
44
|
Nwoye CI, Yu T, Gonzalez C, Seeliger B, Mascagni P, Mutter D, Marescaux J, Padoy N. Rendezvous: attention mechanisms for the recognition of surgical action triplets in endoscopic videos. Med Image Anal 2022; 78:102433. [DOI: 10.1016/j.media.2022.102433] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/20/2021] [Revised: 02/25/2022] [Accepted: 03/21/2022] [Indexed: 10/18/2022]
|
45
|
Huber M, Ourselin S, Bergeles C, Vercauteren T. Deep homography estimation in dynamic surgical scenes for laparoscopic camera motion extraction. COMPUTER METHODS IN BIOMECHANICS AND BIOMEDICAL ENGINEERING. IMAGING & VISUALIZATION 2022; 10:321-329. [PMID: 38013837 PMCID: PMC10478259 DOI: 10.1080/21681163.2021.2002195] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 10/27/2021] [Accepted: 11/01/2021] [Indexed: 11/29/2023]
Abstract
Current laparoscopic camera motion automation relies on rule-based approaches or only focuses on surgical tools. Imitation Learning (IL) methods could alleviate these shortcomings, but have so far been applied to oversimplified setups. Instead of extracting actions from oversimplified setups, in this work we introduce a method that allows to extract a laparoscope holder's actions from videos of laparoscopic interventions. We synthetically add camera motion to a newly acquired dataset of camera motion free da Vinci surgery image sequences through a novel homography generation algorithm. The synthetic camera motion serves as a supervisory signal for camera motion estimation that is invariant to object and tool motion. We perform an extensive evaluation of state-of-the-art (SOTA) Deep Neural Networks (DNNs) across multiple compute regimes, finding our method transfers from our camera motion free da Vinci surgery dataset to videos of laparoscopic interventions, outperforming classical homography estimation approaches in both, precision by 41 % , and runtime on a CPU by 43 % .
Collapse
Affiliation(s)
- Martin Huber
- School of Biomedical Engineering & Image Sciences, Faculty of Life Sciences & Medicine, King’s College London, London, UK
| | - Sébastien Ourselin
- School of Biomedical Engineering & Image Sciences, Faculty of Life Sciences & Medicine, King’s College London, London, UK
| | - Christos Bergeles
- School of Biomedical Engineering & Image Sciences, Faculty of Life Sciences & Medicine, King’s College London, London, UK
| | - Tom Vercauteren
- School of Biomedical Engineering & Image Sciences, Faculty of Life Sciences & Medicine, King’s College London, London, UK
| |
Collapse
|
46
|
Junger D, Frommer SM, Burgert O. State-of-the-art of situation recognition systems for intraoperative procedures. Med Biol Eng Comput 2022; 60:921-939. [PMID: 35178622 PMCID: PMC8933302 DOI: 10.1007/s11517-022-02520-4] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/16/2020] [Accepted: 01/30/2022] [Indexed: 11/05/2022]
Abstract
One of the key challenges for automatic assistance is the support of actors in the operating room depending on the status of the procedure. Therefore, context information collected in the operating room is used to gain knowledge about the current situation. In literature, solutions already exist for specific use cases, but it is doubtful to what extent these approaches can be transferred to other conditions. We conducted a comprehensive literature research on existing situation recognition systems for the intraoperative area, covering 274 articles and 95 cross-references published between 2010 and 2019. We contrasted and compared 58 identified approaches based on defined aspects such as used sensor data or application area. In addition, we discussed applicability and transferability. Most of the papers focus on video data for recognizing situations within laparoscopic and cataract surgeries. Not all of the approaches can be used online for real-time recognition. Using different methods, good results with recognition accuracies above 90% could be achieved. Overall, transferability is less addressed. The applicability of approaches to other circumstances seems to be possible to a limited extent. Future research should place a stronger focus on adaptability. The literature review shows differences within existing approaches for situation recognition and outlines research trends. Applicability and transferability to other conditions are less addressed in current work.
Collapse
Affiliation(s)
- D Junger
- School of Informatics, Research Group Computer Assisted Medicine (CaMed), Reutlingen University, Alteburgstr. 150, 72762, Reutlingen, Germany.
| | - S M Frommer
- School of Informatics, Research Group Computer Assisted Medicine (CaMed), Reutlingen University, Alteburgstr. 150, 72762, Reutlingen, Germany
| | - O Burgert
- School of Informatics, Research Group Computer Assisted Medicine (CaMed), Reutlingen University, Alteburgstr. 150, 72762, Reutlingen, Germany
| |
Collapse
|
47
|
Liu W, Wu Y, Hong Y, Zhang Z, Yue Y, Zhang J. Applications of machine learning in computational nanotechnology. NANOTECHNOLOGY 2022; 33:162501. [PMID: 34965514 DOI: 10.1088/1361-6528/ac46d7] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/05/2021] [Accepted: 12/28/2021] [Indexed: 06/14/2023]
Abstract
Machine learning (ML) has gained extensive attention in recent years due to its powerful data analysis capabilities. It has been successfully applied to many fields and helped the researchers to achieve several major theoretical and applied breakthroughs. Some of the notable applications in the field of computational nanotechnology are ML potentials, property prediction, and material discovery. This review summarizes the state-of-the-art research progress in these three fields. ML potentials bridge the efficiency versus accuracy gap between density functional calculations and classical molecular dynamics. For property predictions, ML provides a robust method that eliminates the need for repetitive calculations for different simulation setups. Material design and drug discovery assisted by ML greatly reduce the capital and time investment by orders of magnitude. In this perspective, several common ML potentials and ML models are first introduced. Using these state-of-the-art models, developments in property predictions and material discovery are overviewed. Finally, this paper was concluded with an outlook on future directions of data-driven research activities in computational nanotechnology.
Collapse
Affiliation(s)
- Wenxiang Liu
- Key Laboratory of Hydraulic Machinery Transients (MOE), School of Power and Mechanical Engineering, Wuhan University, Wuhan, Hubei 430072, People's Republic of China
| | - Yongqiang Wu
- Weichai Power CO., Ltd, Weifang 261061, People's Republic of China
| | - Yang Hong
- Research Computing, RCAC, Purdue University, West Lafayette, IN 47907, United States of America
| | - Zhongtao Zhang
- Holland Computing Center, University of Nebraska-Lincoln, Lincoln, NE, United States of America
| | - Yanan Yue
- Key Laboratory of Hydraulic Machinery Transients (MOE), School of Power and Mechanical Engineering, Wuhan University, Wuhan, Hubei 430072, People's Republic of China
| | - Jingchao Zhang
- NVIDIA AI Technology Center (NVAITC), Santa Clara, CA 95051, United States of America
| |
Collapse
|
48
|
Uncharted Waters of Machine and Deep Learning for Surgical Phase Recognition in Neurosurgery. World Neurosurg 2022; 160:4-12. [PMID: 35026457 DOI: 10.1016/j.wneu.2022.01.020] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/02/2021] [Revised: 01/05/2022] [Accepted: 01/05/2022] [Indexed: 12/20/2022]
Abstract
Recent years have witnessed artificial intelligence (AI) make meteoric leaps in both medicine and surgery, bridging the gap between the capabilities of humans and machines. Digitization of operating rooms and the creation of massive quantities of data have paved the way for machine learning and computer vision applications in surgery. Surgical phase recognition (SPR) is a newly emerging technology that uses data derived from operative videos to train machine and deep learning algorithms to identify the phases of surgery. Advancement of this technology will be key in establishing context-aware surgical systems in the future. By automatically recognizing and evaluating the current surgical scenario, these intelligent systems are able to provide intraoperative decision support, improve operating room efficiency, assess surgical skills, and aid in surgical training and education. Still in its infancy, SPR has been mainly studied in laparoscopic surgeries, with a contrasting stark lack of research within neurosurgery. Given the high-tech and rapidly advancing nature of neurosurgery, we believe SPR has a tremendous untapped potential in this field. Herein, we present an overview of the SPR technology, its potential applications in neurosurgery, and the challenges that lie ahead.
Collapse
|
49
|
Kitaguchi D, Takeshita N, Hasegawa H, Ito M. Artificial intelligence-based computer vision in surgery: Recent advances and future perspectives. Ann Gastroenterol Surg 2022; 6:29-36. [PMID: 35106412 PMCID: PMC8786689 DOI: 10.1002/ags3.12513] [Citation(s) in RCA: 17] [Impact Index Per Article: 8.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/26/2021] [Revised: 09/16/2021] [Accepted: 09/18/2021] [Indexed: 12/04/2022] Open
Abstract
Technology has advanced surgery, especially minimally invasive surgery (MIS), including laparoscopic surgery and robotic surgery. It has led to an increase in the number of technologies in the operating room. They can provide further information about a surgical procedure, e.g. instrument usage and trajectories. Among these surgery-related technologies, the amount of information extracted from a surgical video captured by an endoscope is especially great. Therefore, the automation of data analysis is essential in surgery to reduce the complexity of the data while maximizing its utility to enable new opportunities for research and development. Computer vision (CV) is the field of study that deals with how computers can understand digital images or videos and seeks to automate tasks that can be performed by the human visual system. Because this field deals with all the processes of real-world information acquisition by computers, the terminology "CV" is extensive, and ranges from hardware for image sensing to AI-based image recognition. AI-based image recognition for simple tasks, such as recognizing snapshots, has advanced and is comparable to humans in recent years. Although surgical video recognition is a more complex and challenging task, if we can effectively apply it to MIS, it leads to future surgical advancements, such as intraoperative decision-making support and image navigation surgery. Ultimately, automated surgery might be realized. In this article, we summarize the recent advances and future perspectives of AI-related research and development in the field of surgery.
Collapse
Affiliation(s)
- Daichi Kitaguchi
- Surgical Device Innovation OfficeNational Cancer Center Hospital EastKashiwaJapan
| | - Nobuyoshi Takeshita
- Surgical Device Innovation OfficeNational Cancer Center Hospital EastKashiwaJapan
| | - Hiro Hasegawa
- Surgical Device Innovation OfficeNational Cancer Center Hospital EastKashiwaJapan
| | - Masaaki Ito
- Surgical Device Innovation OfficeNational Cancer Center Hospital EastKashiwaJapan
| |
Collapse
|
50
|
Xu H, Han T, Wang H, Liu S, Hou G, Sun L, Jiang G, Yang F, Wang J, Deng K, Zhou J. OUP accepted manuscript. Eur J Cardiothorac Surg 2022; 62:6555788. [PMID: 35352106 PMCID: PMC9615432 DOI: 10.1093/ejcts/ezac154] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/15/2021] [Revised: 02/10/2022] [Accepted: 03/11/2011] [Indexed: 11/15/2022] Open
Affiliation(s)
- Hao Xu
- Department of Thoracic Surgery, Peking University People's Hospital, Beijing, China
| | - Tingxuan Han
- Center for Statistical Science & Department of Industrial Engineering, Tsinghua University, Beijing, China
| | - Haifeng Wang
- Center for Statistical Science & Department of Industrial Engineering, Tsinghua University, Beijing, China
| | - Shanggui Liu
- Department of Thoracic Surgery, Peking University People's Hospital, Beijing, China
| | - Guanghao Hou
- Department of Thoracic Surgery, Peking University People's Hospital, Beijing, China
| | - Lina Sun
- Central operating Theatre, Peking University People's Hospital, Beijing, China
| | - Guanchao Jiang
- Department of Thoracic Surgery, Peking University People's Hospital, Beijing, China
| | - Fan Yang
- Department of Thoracic Surgery, Peking University People's Hospital, Beijing, China
| | - Jun Wang
- Department of Thoracic Surgery, Peking University People's Hospital, Beijing, China
- Corresponding authors: Department of Thoracic Surgery, Peking University People's Hospital, Beijing 100044, China. Tel: +86 010-88326650; e-mail: (Dr. Jian Zhou); Department of Thoracic Surgery, Peking University People's Hospital, Beijing 100044, China. Tel: 010-88326652; e-mail: (Dr. Jun Wang); Center for Statistical Science & Department of Industrial Engineering, Tsinghua University, Beijing 100084, China. Tel: +86 010-62782453; e-mail: (Ke Deng)
| | - Ke Deng
- Center for Statistical Science & Department of Industrial Engineering, Tsinghua University, Beijing, China
- Corresponding authors: Department of Thoracic Surgery, Peking University People's Hospital, Beijing 100044, China. Tel: +86 010-88326650; e-mail: (Dr. Jian Zhou); Department of Thoracic Surgery, Peking University People's Hospital, Beijing 100044, China. Tel: 010-88326652; e-mail: (Dr. Jun Wang); Center for Statistical Science & Department of Industrial Engineering, Tsinghua University, Beijing 100084, China. Tel: +86 010-62782453; e-mail: (Ke Deng)
| | - Jian Zhou
- Department of Thoracic Surgery, Peking University People's Hospital, Beijing, China
- Corresponding authors: Department of Thoracic Surgery, Peking University People's Hospital, Beijing 100044, China. Tel: +86 010-88326650; e-mail: (Dr. Jian Zhou); Department of Thoracic Surgery, Peking University People's Hospital, Beijing 100044, China. Tel: 010-88326652; e-mail: (Dr. Jun Wang); Center for Statistical Science & Department of Industrial Engineering, Tsinghua University, Beijing 100084, China. Tel: +86 010-62782453; e-mail: (Ke Deng)
| |
Collapse
|