Use of cognitive task analysis to guide the development of performance-based assessments for intraoperative decision making

AI Versus MD: Evaluating the surgical decision-making accuracy of ChatGPT-4

BACKGROUND

ChatGPT-4 is a large language model with possible applications to surgery education The aim of this study was to investigate the accuracy of ChatGPT-4's surgical decision-making compared with general surgery residents and attending surgeons.

METHODS

Five clinical scenarios were created from actual patient data based on common general surgery diagnoses. Scripts were developed to sequentially provide clinical information and ask decision-making questions. Responses to the prompts were scored based on a standardized rubric for a total of 50 points. Each clinical scenario was run through Chat GPT-4 and sent electronically to all general surgery residents and attendings at a single institution. Scores were compared using Wilcoxon rank sum tests.

RESULTS

On average, ChatGPT-4 scored 39.6 points (79.2%, standard deviation ± 0.89 points). A total of five junior residents, 12 senior residents, and five attendings completed the clinical scenarios (resident response rate = 15.9%; attending response rate = 13.8%). On average, the junior residents scored a total of 33.4 (66.8%, standard deviation ± 3.29), senior residents 38.0 (76.0%, standard deviation ± 4.75), and attendings 38.8 (77.6%, standard deviation ± 5.45). ChatGPT-4 scored significantly better than junior residents (P = .009) but was not significantly different from senior residents or attendings. ChatGPT-4 was significantly better than junior residents at identifying the correct operation to perform (P = .0182) and recommending additional workup for postoperative complications (P = .012).

CONCLUSION

ChatGPT-4 performed superior to junior residents and equivalent to senior residents and attendings when faced with surgical patient scenarios. Large language models, such as ChatGPT, may have the potential to be an educational resource for junior residents to develop surgical decision-making skills.

Defining Cognitive Competencies for Breast Augmentation Surgery

BACKGROUND

Breast augmentation is the most performed aesthetic procedure in the United States yet one that surgical trainees have limited exposure to. This creates a lack of confidence in performing this key procedure among graduates. It is imperative to develop novel curricula and objective measures to standardize acquiring competency.

OBJECTIVE

This qualitative study establishes various cognitive competencies and pitfalls in augmentation mammoplasty.

METHODS

Using a priori established task analysis, literary sources and operative observations, a total of 20 cognitive vignettes were developed to conduct cognitive task analyses (CTA) for breast augmentation through semistructured interviews of experts. Interviews were itemized, and verbal data were recorded, transcribed verbatim, and thematically analyzed by reviewers.

RESULTS

Eight experts were interviewed (median age 39 years, 87.5% males, with a median of 7 years in practice). A conceptual framework for breast augmentation was developed and divided into 5 operative stages containing 208 competencies and 41 pitfalls. Pitfalls were mapped to deficits in shared decision making, proper informed consent, prospective hemostasis, and awareness of anatomical landmarks and markings.

CONCLUSIONS

This work provided an inclusive framework of cognitive competencies in breast augmentation surgery to facilitate their assessment. This model guides the analysis of other procedures to transfer cognitive competencies to learners. In a transition toward competency-based education, this provides a primer to assessments that include all aspects of a surgeon's skill set.

Understanding How Experts Do It: A Conceptual Framework for the Open Transversus Abdominis Release Procedure

BACKGROUND

The safe and effective performance of a posterior component separation via a transversus abdominis release (TAR) requires intraoperative judgement and decision-making skills that are difficult to define, standardize, and teach. We herein present the first qualitative study which builds a framework upon which training and objective evaluation of a TAR can be based.

METHODS

Hierarchical and cognitive task analyses for a TAR procedure were performed using semistructured interviews of hernia experts to describe the thoughts and behaviors that exemplify optimal performance. Verbal data was recorded, transcribed, coded, and thematically analyzed.

RESULTS

A conceptual framework was synthesized based on literary sources (4 book chapters, 4 peer-reviewed articles, 3 online videos), 2 field observations, and interviews of 4 hernia experts [median 66 minutes (44-78)]. Subject matter experts practiced a median of 6.5 years (1.5-16) and have completed a median of 300 (60-500) TARs. After 5 rounds of inductive analysis, 80 subtasks, 86 potential errors, 36 cognitive behaviors, and 17 decision points were identified and categorized into 10 procedural steps (midline laparotomy, adhesiolysis, retrorectus dissection, etc.) and 9 fundamental principles: patient physiology and disease burden; tactical modification; tissue reconstruction and wound healing; task completion; choice of technique and instruments; safe planes and danger zones; exposure, ergonomics, environmental limitations; anticipation and forward planning; and tissue trauma and handling.

CONCLUSION

This is the first study to define the key tasks, decisions, and cognitive behaviors that are essential to a successful TAR procedure.

Do Individual Surgeon Preferences Affect Procedural Outcomes?

OBJECTIVES

Surgeon preferences such as instrument and suture selection and idiosyncratic approaches to individual procedure steps have been largely viewed as minor differences in the surgical workflow. We hypothesized that idiosyncratic approaches could be quantified and shown to have measurable effects on procedural outcomes.

METHODS

At the American College of Surgeons (ACS) Clinical Congress, experienced surgeons volunteered to wear motion tracking sensors and be videotaped while evaluating a loop of porcine intestines to identify and repair 2 preconfigured, standardized enterotomies. Video annotation was used to identify individual surgeon preferences and motion data was used to quantify surgical actions. χ 2 analysis was used to determine whether surgical preferences were associated with procedure outcomes (bowel leak).

RESULTS

Surgeons' (N=255) preferences were categorized into 4 technical decisions. Three out of the 4 technical decisions (repaired injuries together, double-layer closure, corner-stitches vs no corner-stitches) played a significant role in outcomes, P <0.05. Running versus interrupted did not affect outcomes. Motion analysis revealed significant differences in average operative times (leak: 6.67 min vs no leak: 8.88 min, P =0.0004) and work effort (leak-path length=36.86 cm vs no leak-path length=49.99 cm, P =0.001). Surgeons who took the riskiest path but did not leak had better bimanual dexterity (leak=0.21/1.0 vs no leak=0.33/1.0, P =0.047) and placed more sutures during the repair (leak=4.69 sutures vs no leak=6.09 sutures, P =0.03).

CONCLUSIONS

Our results show that individual preferences affect technical decisions and play a significant role in procedural outcomes. Future analysis in more complex procedures may make major contributions to our understanding of contributors to procedure outcomes.

Virtual Simulation of Intra-operative Decision-Making for Open Abdominal Aortic Aneurysm Repair: A Mixed Methods Analysis

OBJECTIVE

To create and pilot test a novel open abdominal aortic aneurysm (AAA) repair virtual simulation focused on intraoperative decision-making. To identify if the simulation replicated real-time intra-operative decision-making and discover how learners' respond to this type of simulation.

DESIGN

An explanatory sequential mixed methods study. We developed a step-by-step outline of major intra-operative decision points within a standard open AAA repair. Perioperative and intraoperative decision-making trees were developed and coded into an online virtual simulation. The simulation was piloted. Quantitative data was collected from the simulation platform. We then performed a qualitative thematic analysis on feedback from interviewed participants.

SETTING

Four academic general and vascular surgical training programs across the US.

PARTICIPANTS

Seventeen vascular and general surgery trainees and 6 vascular surgery faculty.

RESULTS

Participants spent on average 27 minutes (range: 8-45 minutes) interacting with the interface. 93% of participants reported feeling they were making real intraoperative decisions. 85% said it added to their knowledge base. 96% requested additional simulations. 22 interviews were completed: 241 primary codes were collapsed into 21 parent codes, and 6 emerging themes identified. Themes included the benefit of how (1) "Virtual Learning Could Standardize the Training Experience"; how (2) "Dealing with the Unexpected" as a trainee is an important part of surgical education growth, and that this (3) "Choose Your Own Adventure" virtual format simulates this intraoperative growth experience. Participants requested a (4) "Looping Feature Feedback Diagram" for future simulation iterations and highlighted that (5) "Fancier is Not Necessarily More Educational." Finally, many trainees wondered about (6) "The Attending Impact" from the simulation: if faculty would notice a difference between trainees who did vs did not utilize the simulation for case preparation.

CONCLUSIONS

Operative simulation training should focus on both technical skills and intra-operative decision-making, particularly "dealing with the unexpected." The learners' responses indicate that a low-fidelity, scalable, virtual platform can effectively deliver knowledge and allow for intra-operative decision-making practice in a remote learning environment.

Teaching and Assessing Cognitive Competencies in Aesthetic and Plastic Surgery

BACKGROUND

Cognition, part of surgical competence, employs forward planning, error prevention, and orchestrating technical skills. Thus, an understanding of mental processes used by experts throughout patient care is essential to teaching such competencies. This study aims to analyze and compare mental models of two distinct procedures in plastic surgery, breast augmentation and flexor tendon repair, to develop a framework to define cognitive competencies in plastic surgery aided by a review of the literature.

METHODS

Based on data from a-priori cognitive task analyses, literary sources, and field observations of breast augmentation surgery and flexor tendon repair, task-lists were produced for each procedure. Two mental models were developed using fuzzy logic cognitive maps to visually illustrate and analyze cognitive processes employed in either procedure. A comparison of the models aided by literature was used to define the cognitive competencies employed, identify differences in the decision-making process, and provide a guiding framework for understanding cognitive competencies.

RESULTS

Five distinct cognitive competency domains were identified from the literature applicable to plastic surgery, namely situation awareness, decision-making, task management, leadership, and communication and teamwork. Differences in processes of decision-making employed between an elective and a trauma setting were identified. A framework to map cognitive competencies within a generic mental model in surgical care was synthesized and methods were suggested for training on such competencies.

CONCLUSION

Cognitive competencies in different settings in plastic surgery are introduced using a comparative study of two mental models of distinct procedures to guide the teaching and assessment of such competencies.

Correlating Personal Resourcefulness and Psychomotor Skills: An Analysis of Stress, Visual Attention and Technical Metrics

Modern surgical education is focused on making use of the available technologies in order to train and assess surgical skill acquisition. Innovative technologies for the automatic, objective assessment of nontechnical skills are currently under research. The main aim of this study is to determine whether personal resourcefulness can be assessed by monitoring parameters that are related to stress and visual attention and whether there is a relation between these and psychomotor skills in surgical education. For this purpose, we implemented an application in order to monitor the electrocardiogram (ECG), galvanic skin response (GSR), gaze and performance of surgeons-in-training while performing a laparoscopic box-trainer task so as to obtain technical and personal resourcefulness' metrics. Eight surgeons (6 nonexperts and 2 experts) completed the experiment. A total of 22 metrics were calculated (7 technical and 15 related to personal resourcefulness) per subject. The average values of these metrics in the presence of stressors were compared with those in their absence and depending on the participants' expertise. The results show that both the mean normalized GSR signal and average surgical instrument's acceleration change significantly when stressors are present. Additionally, the GSR and acceleration were found to be correlated, which indicates that there is a relation between psychomotor skills and personal resourcefulness.

Training Surgeons to Perform Arthroscopic All-Inside Meniscal Repair: A Randomized Controlled Trial Evaluating the Effectiveness of a Novel Cognitive Task Analysis Teaching Tool, Imperial College London/University College London Meniscus Repair Cognitive Task Analysis (IUMeRCTA)

BACKGROUND

All-inside meniscal repair is an increasingly common technique for the surgical treatment of meniscal tears. There are currently no standardized techniques for training residents in this procedure. Cognitive task analysis (CTA) is a method of analyzing and standardizing key steps in a procedure that allows training to be conducted in a validated and reproducible manner.

PURPOSE

(1) To design a digital CTA teaching tool for a standardized all-inside meniscal repair. (2) To evaluate whether CTA-trained residents would perform better in a meniscal repair task compared with a control group who underwent traditional apprenticeship methods of training.

STUDY DESIGN

Controlled laboratory study.

METHODS

Three expert knee surgeons were interviewed using a modified Delphi method to generate a consensus among the ideal technical steps, cognitive decision points, and common errors and solutions for an all-inside meniscal repair. This written information was then combined with visual and audio components and integrated onto a digital platform to create the Imperial College London/University College London Meniscus Repair Cognitive Task Analysis (IUMeRCTA) tool. Eighteen novice residents were randomized into an intervention group (digital CTA tool) and control group (equipment instruction manual). Both groups performed an all-inside meniscal repair on high-fidelity, phantom knee models and were assessed by expert surgeons, blinded to the interventions, using a validated global rating scale (GRS). After a power calculation, median GRS scores were compared between groups using the Mann-Whitney U test; significance was set at P < .05.

RESULTS

For the IUMeRCTA tool design, the procedure was divided into 55 steps across 9 phases: (1) preoperative planning, (2) theater and patient setup, (3) portal placement, (4) meniscal examination, (5) tear reduction, (6) suture planning, (7) suture insertion, (8) repair completion, and (9) postoperative care and rehabilitation. For the trial, the intervention group (mean ± SD GRS, 32 ± 2.9) performed significantly better than did the control group (GRS, 24 ± 3.3; P < .001).

CONCLUSION

This is the first CTA tool to demonstrate objective benefits in training novices to perform an arthroscopic all-inside meniscal repair.

CLINICAL RELEVANCE

The IUMeRCTA tool is an easily accessible and effective adjunct to traditional teaching that enhances learning the all-inside meniscal repair for novice surgeons.

Current applications of artificial intelligence for intraoperative decision support in surgery

Research into medical artificial intelligence (AI) has made significant advances in recent years, including surgical applications. This scoping review investigated AI-based decision support systems targeted at the intraoperative phase of surgery and found a wide range of technological approaches applied across several surgical specialties. Within the twenty-one (n = 21) included papers, three main categories of motivations were identified for developing such technologies: (1) augmenting the information available to surgeons, (2) accelerating intraoperative pathology, and (3) recommending surgical steps. While many of the proposals hold promise for improving patient outcomes, important methodological shortcomings were observed in most of the reviewed papers that made it difficult to assess the clinical significance of the reported performance statistics. Despite limitations, the current state of this field suggests that a number of opportunities exist for future researchers and clinicians to work on AI for surgical decision support with exciting implications for improving surgical care.

Computer Vision Analysis of Intraoperative Video: Automated Recognition of Operative Steps in Laparoscopic Sleeve Gastrectomy

OBJECTIVE(S)

To develop and assess AI algorithms to identify operative steps in laparoscopic sleeve gastrectomy (LSG).

BACKGROUND

Computer vision, a form of artificial intelligence (AI), allows for quantitative analysis of video by computers for identification of objects and patterns, such as in autonomous driving.

METHODS

Intraoperative video from LSG from an academic institution was annotated by 2 fellowship-trained, board-certified bariatric surgeons. Videos were segmented into the following steps: 1) port placement, 2) liver retraction, 3) liver biopsy, 4) gastrocolic ligament dissection, 5) stapling of the stomach, 6) bagging specimen, and 7) final inspection of staple line. Deep neural networks were used to analyze videos. Accuracy of operative step identification by the AI was determined by comparing to surgeon annotations.

RESULTS

Eighty-eight cases of LSG were analyzed. A random 70% sample of these clips was used to train the AI and 30% to test the AI's performance. Mean concordance correlation coefficient for human annotators was 0.862, suggesting excellent agreement. Mean (±SD) accuracy of the AI in identifying operative steps in the test set was 82% ± 4% with a maximum of 85.6%.

CONCLUSIONS

AI can extract quantitative surgical data from video with 85.6% accuracy. This suggests operative video could be used as a quantitative data source for research in intraoperative clinical decision support, risk prediction, or outcomes studies.

A Multicenter Randomized Controlled Trial Evaluating the Effectiveness of Cognitive Training for Anterior Approach Total Hip Arthroplasty

BACKGROUND

For total hip arthroplasty (THA), cognitive training prior to performing real surgery may be an effective adjunct alongside simulation to shorten the learning curve. This study sought to create a cognitive training tool (CTT) to perform anterior approach (AA)-THA, which was validated by expert surgeons, and test its use as a training tool compared with conventional material.

METHODS

We employed a modified Delphi method with 4 expert surgeons from 3 international centers of excellence. Surgeons were independently observed performing THA before undergoing semistructured cognitive task analysis (CTA) and before completing successive rounds of surveys until a consensus was reached. Thirty-six surgical residents (postgraduate year [PGY]-1 through PGY-4) were randomized to cognitive training or training with a standard operation manual with surgical videos before performing a simulated AA-THA.

RESULTS

The consensus CTA defined THA in 11 phases, in which were embedded 46 basic steps, 36 decision points, and 42 critical errors and linked strategies. This CTA was mapped onto an open-access web-based CTT. Surgeons who prepared with the CTT performed a simulated THA 35% more quickly (time, mean 28 versus 38 minutes) with 69% fewer errors in instrument selection (mean 29 versus 49 instances), and required 92% fewer prompts (mean 13 versus 25 instances). They were more accurate in acetabular cup orientation (inclination error, mean 8° versus 10°; anteversion error, mean 14° versus 22°).

CONCLUSIONS

This validated CTT for arthroplasty provides structure for competency-based learning. It is more effective at preparing orthopaedic trainees for a complex procedure than conventional materials, as well as for learning sequence, instrumentation utilization, and motor skills.

CLINICAL RELEVANCE

Cognitive training combines education on decision-making, knowledge, and technical skill. It is an inexpensive technique to teach surgeons to perform hip arthroplasty and is more effective than current preparation methods.

In Search of Characterizing Surgical Skill

OBJECTIVE

This paper provides a literature review and detailed discussion of surgical skill terminology. Culminating in a novel model that proposes a set of unique definitions, this review is designed to facilitate shared understanding to study and develop metrics quantifying surgical skill.

DESIGN

Objective surgical skill analysis depends on consistent definitions and shared understanding of terms like performance, expertise, experience, aptitude, ability, competency, and proficiency.

STRUCTURE

Each term is discussed in turn, drawing from existing literature and colloquial uses.

IMPLICATIONS

A new model of definitions is proposed to cement a common and consistent lexicon for future skills analysis, and to quantitatively describe a surgeon's performance throughout their career.

Surgical procedural map scoring for decision-making in laparoscopic cholecystectomy

INTRODUCTION

The objective of this study was to determine whether decision-based procedural mapping demonstrates differences in attendings versus residents.

METHODS

Attendings and residents were interviewed about operative decision-making in laparoscopic cholecystectomy (LC) using a cognitive task analysis framework. Interviews were converted into procedural maps. Operative steps, patient factors, and surgeon factors noted by attendings and residents were compared. Two scoring methods were used to compare map structures of attendings versus residents.

RESULTS

Six attendings and six residents were interviewed. There were no significant differences in the number of patient or surgeon factors identified. Attendings had significantly more operative steps (29.67 ± 1.9 vs. 23.3 ± 1.9, p = 0.04) and crosslinks (3.2 ± 0.5 vs. 1 ± 0.4, p = 0.005) in their maps and a higher total score (90.2 ± 8.4 vs. 63.2 ± 3.8, p = 0.015) than residents.

CONCLUSION

LC procedural map scoring for attendings and residents demonstrated significant differences in structural complexity and may provide a useful framework for assessing decision making.

Simulation Research in Gastrointestinal and Urologic Care-Challenges and Opportunities: Summary of a National Institute of Diabetes and Digestive and Kidney Diseases and National Institute of Biomedical Imaging and Bioengineering Workshop

: A workshop on "Simulation Research in Gastrointestinal and Urologic Care: Challenges and Opportunities" was held at the National Institutes of Health in June 2016. The purpose of the workshop was to examine the extent to which simulation approaches have been used by skilled proceduralists (not trainees) caring for patients with gastrointestinal and urologic diseases. The current status of research findings in the use and effectiveness of simulation applications was reviewed, and numerous knowledge gaps and research needs were identified by the faculty and the attendees. The paradigm of "deliberate practice," rather than mere repetition, and the value of coaching by experts was stressed by those who have adopted simulation in music and sports. Models that are most useful for the adoption of simulation by expert clinicians have yet to be fully validated. Initial studies on the impact of simulation on safety and error reduction have demonstrated its value in the training domain, but the role of simulation as a strategy for increased procedural safety remains uncertain in the world of the expert practitioner. Although the basic requirements for experienced physicians to acquire new skills have been explored, the widespread availability of such resources is an unrealized goal, and there is a need for well-designed outcome studies to establish the role of simulation in improving the quality of health care.

Understanding complex clinical reasoning in infectious diseases for improving clinical decision support design

Background

Clinical experts’ cognitive mechanisms for managing complexity have implications for the design of future innovative healthcare systems. The purpose of the study is to examine the constituents of decision complexity and explore the cognitive strategies clinicians use to control and adapt to their information environment.

Methods

We used Cognitive Task Analysis (CTA) methods to interview 10 Infectious Disease (ID) experts at the University of Utah and Salt Lake City Veterans Administration Medical Center. Participants were asked to recall a complex, critical and vivid antibiotic-prescribing incident using the Critical Decision Method (CDM), a type of Cognitive Task Analysis (CTA). Using the four iterations of the Critical Decision Method, questions were posed to fully explore the incident, focusing in depth on the clinical components underlying the complexity. Probes were included to assess cognitive and decision strategies used by participants.

Results

The following three themes emerged as the constituents of decision complexity experienced by the Infectious Diseases experts: 1) the overall clinical picture does not match the pattern, 2) a lack of comprehension of the situation and 3) dealing with social and emotional pressures such as fear and anxiety. All these factors contribute to decision complexity. These factors almost always occurred together, creating unexpected events and uncertainty in clinical reasoning. Five themes emerged in the analyses of how experts deal with the complexity. Expert clinicians frequently used 1) watchful waiting instead of over- prescribing antibiotics, engaged in 2) theory of mind to project and simulate other practitioners’ perspectives, reduced very complex cases into simple 3) heuristics, employed 4) anticipatory thinking to plan and re-plan events and consulted with peers to share knowledge, solicit opinions and 5) seek help on patient cases.

Conclusion

The cognitive strategies to deal with decision complexity found in this study have important implications for design future decision support systems for the management of complex patients.

Electronic supplementary material

The online version of this article (doi:10.1186/s12911-015-0221-z) contains supplementary material, which is available to authorized users.