Surgical procedural map scoring for decision-making in laparoscopic cholecystectomy

A foundation for evaluating the surgical artificial intelligence literature

With increasing growth in applications of artificial intelligence (AI) in surgery, it has become essential for surgeons to gain a foundation of knowledge to critically appraise the scientific literature, commercial claims regarding products, and regulatory and legal frameworks that govern the development and use of AI. This guide offers surgeons a framework with which to evaluate manuscripts that incorporate the use of AI. It provides a glossary of common terms, an overview of prerequisite knowledge to maximize understanding of methodology, and recommendations on how to carefully consider each element of a manuscript to assess the quality of the data on which an algorithm was trained, the appropriateness of the methodological approach, the potential for reproducibility of the experiment, and the applicability to surgical practice, including considerations on generalizability and scalability.

Surgical experience and identification of errors in laparoscopic cholecystectomy

BACKGROUND

Surgical errors are acts or omissions resulting in negative consequences and/or increased operating time. This study describes surgeon-reported errors in laparoscopic cholecystectomy.

METHODS

Intraoperative videos were uploaded and annotated on Touch SurgeryTM Enterprise. Participants evaluated videos for severity using a 10-point intraoperative cholecystitis grading score, and errors using Observational Clinical Human Reliability Assessment, which includes skill, consequence, and mechanism classifications.

RESULTS

Nine videos were assessed by 8 participants (3 junior (specialist trainee (ST) 3-5), 2 senior trainees (ST6-8), and 3 consultants). Participants identified 550 errors. Positive relationships were seen between total operating time and error count (r2 = 0.284, P < 0.001), intraoperative grade score and error count (r2 = 0.578, P = 0.001), and intraoperative grade score and total operating time (r2 = 0.157, P < 0.001). Error counts differed significantly across intraoperative phases (H(6) = 47.06, P < 0.001), most frequently at dissection of the hepatocystic triangle (total 282; median 33.5 (i.q.r. 23.5-47.8, range 15-63)), ligation/division of cystic structures (total 124; median 13.5 (i.q.r. 12-19.3, range 10-26)), and gallbladder dissection (total 117; median 14.5 (i.q.r. 10.3-18.8, range 6-26)). There were no significant differences in error counts between juniors, seniors, and consultants (H(2) = 0.03, P = 0.987). Errors were classified differently. For dissection of the hepatocystic triangle, thermal injuries (50 in total) were frequently classified as executional, consequential errors; trainees classified thermal injuries as step done with excessive force, speed, depth, distance, time or rotation (29 out of 50), whereas consultants classified them as incorrect orientation (6 out of 50). For ligation/division of cystic structures, inappropriate clipping (60 errors in total), procedural errors were reported by junior trainees (6 out of 60), but not consultants. For gallbladder dissection, inappropriate dissection (20 errors in total) was reported in incorrect planes by consultants and seniors (6 out of 20), but not by juniors. Poor economy of movement (11 errors in total) was reported more by consultants (8 out of 11) than trainees (3 out of 11).

CONCLUSION

This study suggests that surgical experience influences error interpretation, but the benefits for surgical training are currently unclear.

Education Team Time Out in Oncologic Visceral Surgery Optimizes Surgical Resident Training and Team Communication-Results of a Prospective Trial

BACKGROUND

Surgical education is highly dependent on intraoperative communication. Trainers must know the trainee's training level to ensure high-quality surgical training. A systematic preoperative dialogue (Educational Team Time Out, ETO) was established to discuss the steps of each surgical procedure.

METHODS

Over 6 months, ETO was performed within a time limit of 3 minutes. Digital surveys on the utility of ETO and its impact on performance were conducted immediately after surgery and at the end of the study period among the staff of the participating disciplines (trainer, trainee, surgical nursing staff, anaesthesiologists, and medical students). The number of surgical substeps performed was recorded and compared with the equivalent period one year earlier.

RESULTS

ETO was performed in 64 of the 103 eligible operations (62%). Liver resection (n = 37) was the most frequent procedure, followed by left-sided colorectal surgery (n = 12), partial pancreaticoduodenectomy (n = 6), right-sided hemicolectomies (n = 5), and thyroidectomies (n = 4). Anaesthesiologists most frequently reported that ETO had a direct impact on their work during surgery (90.9%). The influence scores were 46.8% for trainees, 8.8% for trainers, 53.3% for surgical nursing staff and 66.6% for medical students. During the implementation of ETO, a trend towards more assisted substeps in oncologic visceral surgery was seen compared to the corresponding period one year earlier (51% vs.40%; p = 0.11).

CONCLUSION

ETO leads to improved intraoperative communication and more performed substeps during complex procedures, which increases motivation and practical training. This concept can easily be implemented in all surgical specialties to improve surgical education.

Surgical workflow recognition with temporal convolution and transformer for action segmentation

PURPOSE

Automatic surgical workflow recognition enabled by computer vision algorithms plays a key role in enhancing the learning experience of surgeons. It also supports building context-aware systems that allow better surgical planning and decision making which may in turn improve outcomes. Utilizing temporal information is crucial for recognizing context; hence, various recent approaches use recurrent neural networks or transformers to recognize actions.

METHODS

We design and implement a two-stage method for surgical workflow recognition. We utilize R(2+1)D for video clip modeling in the first stage. We propose Action Segmentation Temporal Convolutional Transformer (ASTCFormer) network for full video modeling in the second stage. ASTCFormer utilizes action segmentation transformers (ASFormers) and temporal convolutional networks (TCNs) to build a temporally aware surgical workflow recognition system.

RESULTS

We compare the proposed ASTCFormer with recurrent neural networks, multi-stage TCN, and ASFormer approaches. The comparison is done on a dataset comprised of 207 robotic and laparoscopic cholecystectomy surgical videos annotated for 7 surgical phases. The proposed method outperforms the compared methods achieving a [Formula: see text] relative improvement in the average segmental F1-score over the state-of-the-art ASFormer method. Moreover, our proposed method achieves state-of-the-art results on the publicly available Cholec80 dataset.

CONCLUSION

The improvement in the results when using the proposed method suggests that temporal context could be better captured when adding information from TCN to the ASFormer paradigm. This addition leads to better surgical workflow recognition.

The relationship between the esophageal endoscopic submucosal dissection technical difficulty and its intraoperative process

BACKGROUND

Estimating the esophageal endoscopic submucosal dissection (ESD) technical difficulty is important to reduce complications. Endoscopic duration is one of the related factors to a technical difficulty. The relationship between the esophageal ESD technical difficulty and its intraoperative process was analyzed as a first step toward automatic technical difficulty recognition using artificial intelligence.

METHODS

This study enrolled 75 patients with superficial esophageal cancer who underwent esophageal ESD. The technical difficulty score was established, which consisted of three factors, including total procedure duration, en bloc resection, and complications. Additionally, technical difficulty-related factors, which were perioperative factors that included the intraoperative process, were investigated.

RESULTS

Eight (11%) patients were allocated to high difficulty, whereas 67 patients (89%) were allocated to low difficulty. The intraoperative process, which was shown as the extension of each endoscopic phase, was significantly related to a technical difficulty. The area under the curve (AUC) values were higher at all the phase duration than at the clinical characteristics. Submucosal dissection phase (AUC 0.902; 95% confidence intervals (CI) 0.752-1.000), marking phase (AUC 0.827; 95% CI 0.703-0.951), and early phase which was defined as the duration from the start of marking to the end of submucosal injection (AUC 0.847; 95% CI 0.701-0.992) were significantly related to technical difficulty.

CONCLUSIONS

The intraoperative process, particularly early phase, was strongly associated with esophageal ESD technical difficulty. This study demonstrated the potential for automatic evaluation of esophageal ESD technical difficulty when combined with an AI-based automatic phase evaluation system.

AI-Based Video Segmentation: Procedural Steps or Basic Maneuvers?

INTRODUCTION

Video-based review of surgical procedures has proven to be useful in training by enabling efficiency in the qualitative assessment of surgical skill and intraoperative decision-making. Current video segmentation protocols focus largely on procedural steps. Although some operations are more complex than others, many of the steps in any given procedure involve an intricate choreography of basic maneuvers such as suturing, knot tying, and cutting. The use of these maneuvers at certain procedural steps can convey information that aids in the assessment of the complexity of the procedure, surgical preference, and skill. Our study aims to develop and evaluate an algorithm to identify these maneuvers.

METHODS

A standard deep learning architecture was used to differentiate between suture throws, knot ties, and suture cutting on a data set comprised of videos from practicing clinicians (N = 52) who participated in a simulated enterotomy repair. Perception of the added value to traditional artificial intelligence segmentation was explored by qualitatively examining the utility of identifying maneuvers in a subset of steps for an open colon resection.

RESULTS

An accuracy of 84% was reached in differentiating maneuvers. The precision in detecting the basic maneuvers was 87.9%, 60%, and 90.9% for suture throws, knot ties, and suture cutting, respectively. The qualitative concept mapping confirmed realistic scenarios that could benefit from basic maneuver identification.

CONCLUSIONS

Basic maneuvers can indicate error management activity or safety measures and allow for the assessment of skill. Our deep learning algorithm identified basic maneuvers with reasonable accuracy. Such models can aid in artificial intelligence-assisted video review by providing additional information that can complement traditional video segmentation protocols.

Use of prospective video analysis to understand the impact of technical difficulty on operative process during laparoscopic cholecystectomy

BACKGROUND

An understanding of the impact of operative difficulty on operative process in laparoscopic cholecystectomy is lacking. The aim of the present study was to prospectively analyse digitally recorded laparoscopic cholecystectomy to assess the impact of operative technical difficulty on operative process.

METHODS

Video of laparoscopic cholecystectomy procedures performed at Christchurch Hospital, NZ and North Shore Private Hospital, Sydney Australia were prospectively recorded. Using a framework derived from a previously published standard process video was annotated using a standardized template and stratified by operative grade to evaluate the impact of grade on operative process.

RESULTS

317 patients had their laparoscopic cholecystectomy operations prospectively recorded. Seventy one percent of these videos (n = 225) were annotated. Single ICC of operative grade was 0.760 (0.663-0.842 p < 0.010). Median operative time, rate of operative errors significantly increased and rate of CVS decreased with increasing operative grade. Significant differences in operative anatomy, operative process and instrumentation were seen with increasing grade.

CONCLUSION

Operative technical difficulty is accurately predicted by operative grade and this impacts on operative process with significant implications for both surgeons and patients. Consequently operative grade should be documented routinely as part of a culture of safe laparoscopic cholecystectomy.

Do Individual Surgeon Preferences Affect Procedural Outcomes?

OBJECTIVES

Surgeon preferences such as instrument and suture selection and idiosyncratic approaches to individual procedure steps have been largely viewed as minor differences in the surgical workflow. We hypothesized that idiosyncratic approaches could be quantified and shown to have measurable effects on procedural outcomes.

METHODS

At the American College of Surgeons (ACS) Clinical Congress, experienced surgeons volunteered to wear motion tracking sensors and be videotaped while evaluating a loop of porcine intestines to identify and repair 2 preconfigured, standardized enterotomies. Video annotation was used to identify individual surgeon preferences and motion data was used to quantify surgical actions. χ 2 analysis was used to determine whether surgical preferences were associated with procedure outcomes (bowel leak).

RESULTS

Surgeons' (N=255) preferences were categorized into 4 technical decisions. Three out of the 4 technical decisions (repaired injuries together, double-layer closure, corner-stitches vs no corner-stitches) played a significant role in outcomes, P <0.05. Running versus interrupted did not affect outcomes. Motion analysis revealed significant differences in average operative times (leak: 6.67 min vs no leak: 8.88 min, P =0.0004) and work effort (leak-path length=36.86 cm vs no leak-path length=49.99 cm, P =0.001). Surgeons who took the riskiest path but did not leak had better bimanual dexterity (leak=0.21/1.0 vs no leak=0.33/1.0, P =0.047) and placed more sutures during the repair (leak=4.69 sutures vs no leak=6.09 sutures, P =0.03).

CONCLUSIONS

Our results show that individual preferences affect technical decisions and play a significant role in procedural outcomes. Future analysis in more complex procedures may make major contributions to our understanding of contributors to procedure outcomes.

Artificial Intelligence in Colorectal Cancer Surgery: Present and Future Perspectives

Artificial intelligence (AI) and computer vision (CV) are beginning to impact medicine. While evidence on the clinical value of AI-based solutions for the screening and staging of colorectal cancer (CRC) is mounting, CV and AI applications to enhance the surgical treatment of CRC are still in their early stage. This manuscript introduces key AI concepts to a surgical audience, illustrates fundamental steps to develop CV for surgical applications, and provides a comprehensive overview on the state-of-the-art of AI applications for the treatment of CRC. Notably, studies show that AI can be trained to automatically recognize surgical phases and actions with high accuracy even in complex colorectal procedures such as transanal total mesorectal excision (TaTME). In addition, AI models were trained to interpret fluorescent signals and recognize correct dissection planes during total mesorectal excision (TME), suggesting CV as a potentially valuable tool for intraoperative decision-making and guidance. Finally, AI could have a role in surgical training, providing automatic surgical skills assessment in the operating room. While promising, these proofs of concept require further development, validation in multi-institutional data, and clinical studies to confirm AI as a valuable tool to enhance CRC treatment.

Virtual Simulation of Intra-operative Decision-Making for Open Abdominal Aortic Aneurysm Repair: A Mixed Methods Analysis

OBJECTIVE

To create and pilot test a novel open abdominal aortic aneurysm (AAA) repair virtual simulation focused on intraoperative decision-making. To identify if the simulation replicated real-time intra-operative decision-making and discover how learners' respond to this type of simulation.

DESIGN

An explanatory sequential mixed methods study. We developed a step-by-step outline of major intra-operative decision points within a standard open AAA repair. Perioperative and intraoperative decision-making trees were developed and coded into an online virtual simulation. The simulation was piloted. Quantitative data was collected from the simulation platform. We then performed a qualitative thematic analysis on feedback from interviewed participants.

SETTING

Four academic general and vascular surgical training programs across the US.

PARTICIPANTS

Seventeen vascular and general surgery trainees and 6 vascular surgery faculty.

RESULTS

Participants spent on average 27 minutes (range: 8-45 minutes) interacting with the interface. 93% of participants reported feeling they were making real intraoperative decisions. 85% said it added to their knowledge base. 96% requested additional simulations. 22 interviews were completed: 241 primary codes were collapsed into 21 parent codes, and 6 emerging themes identified. Themes included the benefit of how (1) "Virtual Learning Could Standardize the Training Experience"; how (2) "Dealing with the Unexpected" as a trainee is an important part of surgical education growth, and that this (3) "Choose Your Own Adventure" virtual format simulates this intraoperative growth experience. Participants requested a (4) "Looping Feature Feedback Diagram" for future simulation iterations and highlighted that (5) "Fancier is Not Necessarily More Educational." Finally, many trainees wondered about (6) "The Attending Impact" from the simulation: if faculty would notice a difference between trainees who did vs did not utilize the simulation for case preparation.

CONCLUSIONS

Operative simulation training should focus on both technical skills and intra-operative decision-making, particularly "dealing with the unexpected." The learners' responses indicate that a low-fidelity, scalable, virtual platform can effectively deliver knowledge and allow for intra-operative decision-making practice in a remote learning environment.

Colorectal cancer: Artificial intelligence and its role in surgical decision making

Despite several advances in the oncological management of colorectal cancer (CRC), there still remains a lacuna in the treatment strategy, which differs from center to center and on the philosophy of the treating clinician that is not without bias. Personalized treatment is essential for the treatment of CRC to achieve better long-term outcomes and to reduce morbidity. Surgery has an important role to play in the treatment. Surgical treatment of CRC is decided based on clinical parameters and investigations and hence likely to have judgmental errors. Artificial intelligence has been reported to be useful in the surveillance, diagnosis, treatment, and follow-up with accuracy in several malignancies. However, it is still evolving and yet to be established in surgical decision making in CRC. It is not only useful preoperatively but also intraoperatively. Artificial intelligence helps to rectify the human surgical decision when clinical data and radiological and laboratory parameters are fed into the computer and may guide correct surgical treatment.

The use of cognitive task analysis in clinical and health services research — a systematic review

Background

At times, clinical case complexity and different types of uncertainty present challenges to less experienced clinicians or the naive application of clinical guidelines where this may not be appropriate. Cognitive task analysis (CTA) methods are used to elicit, document and transfer tacit knowledge about how experts make decisions.

Methods

We conducted a methodological review to describe the use of CTA methods in understanding expert clinical decision-making. We searched MEDLINE, EMBASE and PsycINFO from inception to 2019 for primary research studies which described the use of CTA methods to understand how qualified clinicians made clinical decisions in real-world clinical settings.

Results

We included 81 articles (80 unique studies) from 13 countries, published from 1993 to 2019, most commonly from surgical and critical care settings. The most common aims were to understand expert decision-making in particular clinical scenarios, using expert decision-making in the development of training programmes, understanding whether decision support tools were warranted and understanding procedural variability and error identification or reduction. Critical decision method (CDM) and CTA interviews were most frequently used, with hierarchical task analysis, task knowledge structures, think-aloud protocols and other methods less commonly used. Studies used interviews, observation, think-aloud exercises, surveys, focus groups and a range of more CTA-specific methodologies such as the systematic human error reduction and prediction approach. Researchers used CTA methods to investigate routine/typical (n = 64), challenging (n = 13) or more uncommon, rare events and anomalies (n = 3).

Conclusions

In conclusion, the elicitation of expert tacit knowledge using CTA has seen increasing use in clinical specialties working under challenging time pressures, complexity and uncertainty. CTA methods have great potential in the development, refinement, modification or adaptation of complex interventions, clinical protocols and practice guidelines.

Registration

PROSPERO ID CRD42019128418.

Supplementary Information

The online version contains supplementary material available at 10.1186/s40814-022-01002-6.

Impact of the Lack of Diversity Within Surgery Career Pathways and Mitigating Factors

The lack of diversity in surgical career pathways impacts the cultural competence of the learning and working environment, the variety of leadership styles found within surgical leadership, and the ability of an organization to achieve equity in the workplace due to ongoing mistrust and untouched bias. Leading mitigating factors include developing pathways for greater numbers of diverse people at the high school and college level and implicit bias training. Though educators have had some success with these factors in the initial stages of diversifying early pathways, these factors are not yet correlated to entry into a surgical career. Future solutions to the lack of diversity in surgery will be predicated on surgeons collectively valuing justice, equity, diversity, and inclusion.

Dissecting Cardiac Surgery: A Video-based Recall Protocol to Elucidate Team Cognitive Processes in the Operating Room

OBJECTIVE

The aim of this study was to elucidate the cognitive processes involved in surgical procedures from the perspective of different team roles (surgeon, anesthesiologist, and perfusionist) and provide a comprehensive compilation of intraoperative cognitive processes.

SUMMARY BACKGROUND DATA

Nontechnical skills play a crucial role in surgical team performance and understanding the cognitive processes underlying the intraoperative phase of surgery is essential to improve patient safety in the operating room (OR).

METHODS

A mixed-methods approach encompassing semistructured interviews with 9 subject-matter experts. A cognitive task analysis was built upon a hierarchical segmentation of coronary artery bypass grafting procedures and a cued-recall protocol using video vignettes was used.

RESULTS

A total of 137 unique surgical cognitive processes were identified, including 33 decision points, 23 critical communications, 43 pitfalls, and 38 strategies. Self-report cognitive workload varied substantially, depending on team role and surgical step. A web-based dashboard was developed, providing an integrated visualization of team cognitive processes in the OR that allows readers to intuitively interact with the study findings.

CONCLUSIONS

This study advances the current body of knowledge by making explicit relevant cognitive processes involved during the intraoperative phase of cardiac surgery from the perspective of multiple OR team members. By displaying the research findings in an interactive dashboard, we provide trainees with new knowledge in an innovative fashion that could be used to enhance learning outcomes. In addition, the approach used in the present study can be used to deeply understand the cognitive factors underlying surgical adverse events and errors in the OR.

Surgical data science and artificial intelligence for surgical education

Surgical data science (SDS) aims to improve the quality of interventional healthcare and its value through the capture, organization, analysis, and modeling of procedural data. As data capture has increased and artificial intelligence (AI) has advanced, SDS can help to unlock augmented and automated coaching, feedback, assessment, and decision support in surgery. We review major concepts in SDS and AI as applied to surgical education and surgical oncology.

Machine Learning for Surgical Phase Recognition: A Systematic Review

OBJECTIVE

To provide an overview of ML models and data streams utilized for automated surgical phase recognition.

BACKGROUND

Phase recognition identifies different steps and phases of an operation. ML is an evolving technology that allows analysis and interpretation of huge data sets. Automation of phase recognition based on data inputs is essential for optimization of workflow, surgical training, intraoperative assistance, patient safety, and efficiency.

METHODS

A systematic review was performed according to the Cochrane recommendations and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement. PubMed, Web of Science, IEEExplore, GoogleScholar, and CiteSeerX were searched. Literature describing phase recognition based on ML models and the capture of intraoperative signals during general surgery procedures was included.

RESULTS

A total of 2254 titles/abstracts were screened, and 35 full-texts were included. Most commonly used ML models were Hidden Markov Models and Artificial Neural Networks with a trend towards higher complexity over time. Most frequently used data types were feature learning from surgical videos and manual annotation of instrument use. Laparoscopic cholecystectomy was used most commonly, often achieving accuracy rates over 90%, though there was no consistent standardization of defined phases.

CONCLUSIONS

ML for surgical phase recognition can be performed with high accuracy, depending on the model, data type, and complexity of surgery. Different intraoperative data inputs such as video and instrument type can successfully be used. Most ML models still require significant amounts of manual expert annotations for training. The ML models may drive surgical workflow towards standardization, efficiency, and objectiveness to improve patient outcome in the future.

REGISTRATION PROSPERO

CRD42018108907.

OR black box and surgical control tower: Recording and streaming data and analytics to improve surgical care

Effective and safe surgery results from a complex sociotechnical process prone to human error. Acquiring large amount of data on surgical care and modelling the process of surgery with artificial intelligence's computational methods could shed lights on system strengths and limitations and enable computer-based smart assistance. With this vision in mind, surgeons and computer scientists have joined forces in a novel discipline called Surgical Data Science. In this regard, operating room (OR) black boxes and surgical control towers are being developed to systematically capture comprehensive data on surgical procedures and to oversee and assist during operating rooms activities, respectively. Most of the early Surgical Data Science works have focused on understanding risks and resilience factors affecting surgical safety, the context and workflow of procedures, and team behaviors. These pioneering efforts in sensing and analyzing surgical activities, together with the advent of precise robotic actuators, bring surgery on the verge of a fourth revolution characterized by smart assistance in perceptual, cognitive and physical tasks. Barriers to implement this vision exist, but the surgical-technical partnerships set by ambitious efforts such as the OR black box and the surgical control tower are working to overcome these roadblocks and translate the vision and early works described in the manuscript into value for patients, surgeons and health systems.

Computer Vision in the Operating Room: Opportunities and Caveats

Effectiveness of computer vision techniques has been demonstrated through a number of applications, both within and outside healthcare. The operating room environment specifically is a setting with rich data sources compatible with computational approaches and high potential for direct patient benefit. The aim of this review is to summarize major topics in computer vision for surgical domains. The major capabilities of computer vision are described as an aid to surgical teams to improve performance and contribute to enhanced patient safety. Literature was identified through leading experts in the fields of surgery, computational analysis and modeling in medicine, and computer vision in healthcare. The literature supports the application of computer vision principles to surgery. Potential applications within surgery include operating room vigilance, endoscopic vigilance, and individual and team-wide behavioral analysis. To advance the field, we recommend collecting and publishing carefully annotated datasets. Doing so will enable the surgery community to collectively define well-specified common objectives for automated systems, spur academic research, mobilize industry, and provide benchmarks with which we can track progress. Leveraging computer vision approaches through interdisciplinary collaboration and advanced approaches to data acquisition, modeling, interpretation, and integration promises a powerful impact on patient safety, public health, and financial costs.

Current applications of artificial intelligence for intraoperative decision support in surgery

Research into medical artificial intelligence (AI) has made significant advances in recent years, including surgical applications. This scoping review investigated AI-based decision support systems targeted at the intraoperative phase of surgery and found a wide range of technological approaches applied across several surgical specialties. Within the twenty-one (n = 21) included papers, three main categories of motivations were identified for developing such technologies: (1) augmenting the information available to surgeons, (2) accelerating intraoperative pathology, and (3) recommending surgical steps. While many of the proposals hold promise for improving patient outcomes, important methodological shortcomings were observed in most of the reviewed papers that made it difficult to assess the clinical significance of the reported performance statistics. Despite limitations, the current state of this field suggests that a number of opportunities exist for future researchers and clinicians to work on AI for surgical decision support with exciting implications for improving surgical care.

Computer Vision Analysis of Intraoperative Video: Automated Recognition of Operative Steps in Laparoscopic Sleeve Gastrectomy

OBJECTIVE(S)

To develop and assess AI algorithms to identify operative steps in laparoscopic sleeve gastrectomy (LSG).

BACKGROUND

Computer vision, a form of artificial intelligence (AI), allows for quantitative analysis of video by computers for identification of objects and patterns, such as in autonomous driving.

METHODS

Intraoperative video from LSG from an academic institution was annotated by 2 fellowship-trained, board-certified bariatric surgeons. Videos were segmented into the following steps: 1) port placement, 2) liver retraction, 3) liver biopsy, 4) gastrocolic ligament dissection, 5) stapling of the stomach, 6) bagging specimen, and 7) final inspection of staple line. Deep neural networks were used to analyze videos. Accuracy of operative step identification by the AI was determined by comparing to surgeon annotations.

RESULTS

Eighty-eight cases of LSG were analyzed. A random 70% sample of these clips was used to train the AI and 30% to test the AI's performance. Mean concordance correlation coefficient for human annotators was 0.862, suggesting excellent agreement. Mean (±SD) accuracy of the AI in identifying operative steps in the test set was 82% ± 4% with a maximum of 85.6%.

CONCLUSIONS

AI can extract quantitative surgical data from video with 85.6% accuracy. This suggests operative video could be used as a quantitative data source for research in intraoperative clinical decision support, risk prediction, or outcomes studies.