Measuring intra-operative decision-making during laparoscopic cholecystectomy: validity evidence for a novel interactive Web-based assessment tool

Development, deployment and scaling of operating room-ready artificial intelligence for real-time surgical decision support

Deep learning for computer vision can be leveraged for interpreting surgical scenes and providing surgeons with real-time guidance to avoid complications. However, neither generalizability nor scalability of computer-vision-based surgical guidance systems have been demonstrated, especially to geographic locations that lack hardware and infrastructure necessary for real-time inference. We propose a new equipment-agnostic framework for real-time use in operating suites. Using laparoscopic cholecystectomy and semantic segmentation models for predicting safe/dangerous ("Go"/"No-Go") zones of dissection as an example use case, this study aimed to develop and test the performance of a novel data pipeline linked to a web-platform that enables real-time deployment from any edge device. To test this infrastructure and demonstrate its scalability and generalizability, lightweight U-Net and SegFormer models were trained on annotated frames from a large and diverse multicenter dataset from 136 institutions, and then tested on a separate prospectively collected dataset. A web-platform was created to enable real-time inference on any surgical video stream, and performance was tested on and optimized for a range of network speeds. The U-Net and SegFormer models respectively achieved mean Dice scores of 57% and 60%, precision 45% and 53%, and recall 82% and 75% for predicting the Go zone, and mean Dice scores of 76% and 76%, precision 68% and 68%, and recall 92% and 92% for predicting the No-Go zone. After optimization of the client-server interaction over the network, we deliver a prediction stream of at least 60 fps and with a maximum round-trip delay of 70 ms for speeds above 8 Mbps. Clinical deployment of machine learning models for surgical guidance is feasible and cost-effective using a generalizable, scalable and equipment-agnostic framework that lacks dependency on hardware with high computing performance or ultra-fast internet connection speed.

Development of an Artificial Intelligence Tool for Intraoperative Guidance During Endovascular Abdominal Aortic Aneurysm Repair

BACKGROUND

Adverse events during surgery can occur in part due to errors in visual perception and judgment. Deep learning is a branch of artificial intelligence (AI) that has shown promise in providing real-time intraoperative guidance. This study aims to train and test the performance of a deep learning model that can identify inappropriate landing zones during endovascular aneurysm repair (EVAR).

METHODS

A deep learning model was trained to identify a "No-Go" landing zone during EVAR, defined by coverage of the lowest renal artery by the stent graft. Fluoroscopic images from elective EVAR procedures performed at a single institution and from open-access sources were selected. Annotations of the "No-Go" zone were performed by trained annotators. A 10-fold cross-validation technique was used to evaluate the performance of the model against human annotations. Primary outcomes were intersection-over-union (IoU) and F1 score and secondary outcomes were pixel-wise accuracy, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV).

RESULTS

The AI model was trained using 369 images procured from 110 different patients/videos, including 18 patients/videos (44 images) from open-access sources. For the primary outcomes, IoU and F1 were 0.43 (standard deviation ± 0.29) and 0.53 (±0.32), respectively. For the secondary outcomes, accuracy, sensitivity, specificity, NPV, and PPV were 0.97 (±0.002), 0.51 (±0.34), 0.99 (±0.001). 0.99 (±0.002), and 0.62 (±0.34), respectively.

CONCLUSIONS

AI can effectively identify suboptimal areas of stent deployment during EVAR. Further directions include validating the model on datasets from other institutions and assessing its ability to predict optimal stent graft placement and clinical outcomes.

Defining Cognitive Competencies for Breast Augmentation Surgery

BACKGROUND

Breast augmentation is the most performed aesthetic procedure in the United States yet one that surgical trainees have limited exposure to. This creates a lack of confidence in performing this key procedure among graduates. It is imperative to develop novel curricula and objective measures to standardize acquiring competency.

OBJECTIVE

This qualitative study establishes various cognitive competencies and pitfalls in augmentation mammoplasty.

METHODS

Using a priori established task analysis, literary sources and operative observations, a total of 20 cognitive vignettes were developed to conduct cognitive task analyses (CTA) for breast augmentation through semistructured interviews of experts. Interviews were itemized, and verbal data were recorded, transcribed verbatim, and thematically analyzed by reviewers.

RESULTS

Eight experts were interviewed (median age 39 years, 87.5% males, with a median of 7 years in practice). A conceptual framework for breast augmentation was developed and divided into 5 operative stages containing 208 competencies and 41 pitfalls. Pitfalls were mapped to deficits in shared decision making, proper informed consent, prospective hemostasis, and awareness of anatomical landmarks and markings.

CONCLUSIONS

This work provided an inclusive framework of cognitive competencies in breast augmentation surgery to facilitate their assessment. This model guides the analysis of other procedures to transfer cognitive competencies to learners. In a transition toward competency-based education, this provides a primer to assessments that include all aspects of a surgeon's skill set.

Validation of an artificial intelligence platform for the guidance of safe laparoscopic cholecystectomy

BACKGROUND

Many surgical adverse events, such as bile duct injuries during laparoscopic cholecystectomy (LC), occur due to errors in visual perception and judgment. Artificial intelligence (AI) can potentially improve the quality and safety of surgery, such as through real-time intraoperative decision support. GoNoGoNet is a novel AI model capable of identifying safe ("Go") and dangerous ("No-Go") zones of dissection on surgical videos of LC. Yet, it is unknown how GoNoGoNet performs in comparison to expert surgeons. This study aims to evaluate the GoNoGoNet's ability to identify Go and No-Go zones compared to an external panel of expert surgeons.

METHODS

A panel of high-volume surgeons from the SAGES Safe Cholecystectomy Task Force was recruited to draw free-hand annotations on frames of prospectively collected videos of LC to identify the Go and No-Go zones. Expert consensus on the location of Go and No-Go zones was established using Visual Concordance Test pixel agreement. Identification of Go and No-Go zones by GoNoGoNet was compared to expert-derived consensus using mean F1 Dice Score, and pixel accuracy, sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV).

RESULTS

A total of 47 frames from 25 LC videos, procured from 3 countries and 9 surgeons, were annotated simultaneously by an expert panel of 6 surgeons and GoNoGoNet. Mean (± standard deviation) F1 Dice score were 0.58 (0.22) and 0.80 (0.12) for Go and No-Go zones, respectively. Mean (± standard deviation) accuracy, sensitivity, specificity, PPV and NPV for the Go zones were 0.92 (0.05), 0.52 (0.24), 0.97 (0.03), 0.70 (0.21), and 0.94 (0.04) respectively. For No-Go zones, these metrics were 0.92 (0.05), 0.80 (0.17), 0.95 (0.04), 0.84 (0.13) and 0.95 (0.05), respectively.

CONCLUSIONS

AI can be used to identify safe and dangerous zones of dissection within the surgical field, with high specificity/PPV for Go zones and high sensitivity/NPV for No-Go zones. Overall, model prediction was better for No-Go zones compared to Go zones. This technology may eventually be used to provide real-time guidance and minimize the risk of adverse events.

SAGES safe cholecystectomy modules improve practicing surgeons' judgment: results of a randomized, controlled trial

BACKGROUND

Despite the advantages of laparoscopic cholecystectomy, major bile duct injury (BDI) rates during this operation remain unacceptably high. In October 2018, SAGES released the Safe Cholecystectomy modules, which define specific strategies to minimize the risk of BDI. This study aims to investigate whether this curriculum can change the knowledge and behaviors of surgeons in practice.

METHODS

Practicing surgeons were recruited from the membership of SAGES and the American College of Surgeons Advisory Council for Rural Surgery. All participants completed a baseline assessment (pre-test) that involved interpreting cholangiograms, troubleshooting difficult cases, and managing BDI. Participants' dissection strategies during cholecystectomy were also compared to the strategies of a panel of 15 experts based on accuracy scores using the Think Like a Surgeon validated web-based platform. Participants were then randomized to complete the Safe Cholecystectomy modules (Safe Chole module group) or participate in usually scheduled CME activities (control group). Both groups completed repeat assessments (post-tests) one month after randomization.

RESULTS

Overall, 41 participants were eligible for analysis, including 18 Safe Chole module participants and 23 controls. The two groups had no significant differences in pre-test scores. However, at post-test, Safe Chole module participants made significantly fewer errors managing BDI and interpreting cholangiograms. Safe Chole module participants were less likely to convert to an open operation on the post-test than controls when facing challenging dissections. However, Safe Chole module participants displayed a similar incidence of errors when evaluating adequate critical views of safety.

CONCLUSIONS

In this randomized-controlled trial, the SAGES Safe Cholecystectomy modules improved surgeons' abilities to interpret cholangiograms and safely manage BDI. Additionally, surgeons who studied the modules were less likely to convert to open during difficult dissections. These data show the power of the Safe Cholecystectomy modules to affect practicing surgeons' behaviors in a measurable and meaningful way.

Development and evaluation of a virtual knowledge assessment tool for transanal total mesorectal excision

BACKGROUND

Transanal total mesorectal excision (TATME) is difficult to learn and can result in serious complications. Current paradigms for assessing performance and competency may be insufficient. This study aims to develop and provide preliminary validity evidence for a TATME virtual assessment tool (TATME-VAT) to assess the cognitive skills necessary to safely complete TATME dissection.

METHODS

Participants from North America, Europe, Japan and China completed the test via an interactive online platform between 11/2019 and 05/2020. They were grouped into expert, experienced and novice surgeons depending on the number of independently performed TATMEs. TATME-VAT is a 24-item web-based assessment evaluating advanced cognitive skills, designed according to a blueprint from consensus guidelines. Eight items were multiple choice questions. Sixteen items required making annotations on still frames of TATME videos (VCT) and were scored using a validated algorithm derived from experts' responses. Annotation (range 0-100), multiple choice (range 0-100), and overall scores (sum of annotation and multiple-choice scores, normalized to μ = 50 and σ = 10) were reported.

RESULTS

There were significant differences between the expert, experienced, and novice groups for the annotation (p < 0.001), multiple-choice (p < 0.001), and overall scores (p < 0.001). The annotation (p = 0.439) and overall (p = 0.152) scores were similar between the experienced and novice groups. Annotation scores were higher in participants with 51 or more vs. 30-50 vs. less than 30 cases. Scores were also lower in users with a self-reported recent complication vs. those without.

CONCLUSIONS

This study describes the development of an interactive video-based virtual assessment tool for TATME dissection and provides initial validity evidence for its use.

Crisis recovery in surgery: Error management and problem solving in safety-critical situations

BACKGROUND

Surgical crises, both clinical and executive, carry risk of harm to patients, staff, and organizations. Once stabilized and contained, crisis recovery requires complex decision-making and problem-solving to address primary failures (errors) and their consequences. In contrast to other safety-critical professions, surgeons may lack access to crisis recovery strategies and tools that go beyond the technical aspects of clinical practice. This study aims to develop a framework for surgical crisis recovery based on problem-solving interventions used by pilots in commercial aviation.

METHODS

This study undertook observational fieldwork, semistructured interviews, and focus groups with senior airline pilots and health care safety experts. Thematic analysis using the framework method identified key interventions applicable to surgical crisis recovery. Subsequently, expert group consensus adapted and content validated this model for clinical use.

RESULTS

Qualitative data from 22 aviation and health care safety experts informed surgical crisis resolution. This consisted of 3 strategies: (1) building cognitive capacity by improving situational awareness and workload management; (2) using checklists in abnormal situations to implement emergency operating procedures; (3) undertaking structured decision-making using analysis-based problem-solving cycles (eg, T-DODAR framework). Twelve tools were validated and adapted to aid implementation of these strategies.

CONCLUSION

Once stabilized, surgical crises may be resolved using 3 sequential strategies derived from commercial aviation.

Video Context Improves Performance in Identifying Operative Planes on Static Surgical Images

BACKGROUND

Correct identification of the surgical tissue planes of dissection is paramount at the operating room, and the needed skills seem to be improved with realistic dynamic models rather than mere still images. The objective is to assess the role of adding video prequels to still images taken from operations on the precision and accuracy of tissue plane identification using a validated simulation model, considering various levels of surgeons' experience.

METHODS

A prospective observational study was conducted involving 15 surgeons distributed to three equal groups, including a consultant group [C], a senior group [S], and a junior group [J]. Subjects were asked to identify and draw ideal tissue planes in 20 images selected at suitable operative moments of identification before and after showing a 10- second videoclip preceding the still image. A validated comparative metric (using a modified Hausdorff distance [%Hdu] for object matching) was used to measure the distance between lines. A precision analysis was carried out based on the difference in %Hdu between lines drawn before and after watching the videos, and between-group comparisons were analyzed using a one-way analysis of variance (ANOVA). The analysis of accuracy was done on the difference in %Hdu between lines drawn by the subjects and the ideal lines provided by an expert panel. The impact of videos on accuracy was assessed using a repeated-measures ANOVA.

RESULTS

The C group showed the highest preciseness as compared to the S and J groups (mean Hdu 9.17±11.86 versus 12.1±15.5 and 20.0±18.32, respectively, p <0.001) and significant differences between groups were found in 14 images (70%). Considering the expert panel as a reference, the interaction between time and experience level was significant ( F (2, 597) = 4.52, p <0.001). Although the subjects of the J group were significantly less accurate than other surgeons, only this group showed significant improvements in mean %Hdu values after watching the lead-in videos ( F (1, 597) = 6.04, p = 0.014).

CONCLUSIONS

Adding video context improved the ability of junior trainees to identify tissue planes of dissection. A realistic model is recommended considering experience-based differences in precision in training programs.

The long game: Evolution of clinical decision making throughout residency and fellowship

BACKGROUND

The purpose of this study was to explore the trajectory of autonomy in clinical decision making.

METHODS

We conducted a qualitative secondary analysis of interviews with 45 residents and fellows from the General Surgery and Obstetrics & Gynecology departments across all clinical postgraduate years (PGY) using convenience sampling. Each interview was recorded, transcribed and iteratively analyzed using a framework method.

RESULTS

A total of 16 junior residents, 22 senior residents and 7 fellows participated in 12 original interviews. Early in training residents take their abstract ideas about disease processes and make them concrete in their applications to patient care. A transitional stage follows in which residents apply concepts to concrete patient care. Chief residents re-abstract their concrete technical and clinical knowledge to prepare for future surgical practice.

CONCLUSIONS

Understanding where each learner is on this pathway will assist development of curriculum that fosters resident readiness for practice at each PGY level.

SMaRT Assessment Tool: An Innovative Approach for Objective Assessment of Flap Designs

SUMMARY

The teaching and assessment of ideal surgical markings for local flaps required for optimal aesthetic and functional outcomes remain a challenge in the present era of competency-based surgical education. The authors utilized the bilobed flap for nasal reconstruction as a proof of concept for the development of an innovative objective assessment tool based on statistical shape analysis, with a focus on providing automated, evidence-based, objective, specific, and practical feedback to the learner. The proposed tool is based on Procrustes statistical shape analysis, previously used for the assessment of facial asymmetry in plastic surgery. For performance boundary testing, a series of optimal and suboptimal designs generated in deliberate violation of the established ideals of optimal bilobed flap design were evaluated, and a four-component feedback score of Scale, Mismatch, Rotation, and Translation (SMaRT) was generated. The SMaRT assessment tool demonstrated the capacity to proportionally score a spectrum of designs (n = 36) ranging from subtle to significant variations of optimal, with excellent computational and clinically reasonable performance boundaries. In terms of shape mismatch, changes in SMaRT score also correlated with intended violations in designs away from the ideal flap design. This innovative educational approach could aid in incorporating objective feedback in simulation-based platforms in order to facilitate deliberate practice in flap design, with the potential for adoption in other fields of plastic surgery to automate assessment processes.

The evolution of surgical virtual education and telementoring: One surgeon's journey

The first era of the global proliferation of surgical advancements involved surgical infection rate and technique breakthroughs by Lister, Halsted, and others. This was propagated by letters, academic papers, and international visits. While success was achieved, it was at a suboptimal pace. In the current era of minimally invasive surgical approaches, these methods are inadequate. This paper chronicles the development and application of virtual learning and telementoring as force multipliers to speed procedural adoption and proliferation.

Challenges in surgical video annotation

Annotation of surgical video is important for establishing ground truth in surgical data science endeavors that involve computer vision. With the growth of the field over the last decade, several challenges have been identified in annotating spatial, temporal, and clinical elements of surgical video as well as challenges in selecting annotators. In reviewing current challenges, we provide suggestions on opportunities for improvement and possible next steps to enable translation of surgical data science efforts in surgical video analysis to clinical research and practice.

Hanging up the surgical cap: Assessing the competence of aging surgeons

BACKGROUND

As the average age of surgeons continues to rise, determining when a surgeon should retire is an important public safety concern.

AIM

To investigate strategies used to determine competency in the industrial workplace that could be transferrable in the assessment of aging surgeons and to identify existing competency assessments of practicing surgeons.

METHODS

We searched websites describing non-medical professions within the United States where cognitive and physical competency are necessary for public safety. The mandatory age and certification process, including cognitive and physical requirements, were reported for each profession. Methods for determining surgical competency currently in use, and those existing in the literature, were also identified.

RESULTS

Four non-medical professions requiring mental and physical aptitude that involve public safety and have mandatory testing and/or retirement were identified: Airline pilots, air traffic controllers, firefighters, and United States State Judges. Nine late career practitioner policies designed to evaluate the ageing physician, including surgeons, were described. Six of these policies included subjective performance testing, 4 using peer assessment and 2 using dexterity testing. Six objective testing methods for evaluation of surgeon technical skill were identified in the literature. All were validated for surgical trainees. Only Objective Structured Assessment of Technical Skills (OSATS) was capable of distinguishing between surgeons of different skill level and showing a relationship between skill level and post-operative outcomes.

CONCLUSION

A surgeon should not be forced to hang up his/her surgical cap at a predetermined age, but should be able to practice for as long as his/her surgical skills are objectively maintained at the appropriate level of competency. The strategy of using skill-based simulations in evaluating non-medical professionals can be similarly used as part of the assessment of the ageing surgeons' surgical competency, showing who may require remediation or retirement.

Leveraging Videoconferencing Technology to Augment Surgical Training During a Pandemic

Our objective was to review the use of videoconferencing as a practical tool for remote surgical education and to propose a model to overcome the impact of a pandemic on resident training.

Summary Background Data

In response to the coronavirus disease 2019 pandemic, most institutions and residency programs have been restructured to minimize the number of residents in the hospital as well as their interactions with patients and to promote physical distancing measures. This has resulted in decreased resident operative exposure, responsibility, and autonomy, hindering their educational goals and ability to achieve surgical expertise necessary for independent practice.

Methods

We conducted a narrative review to explore the use of videoconferencing for remote broadcasting of surgical procedures, telecoaching using surgical videos, telesimulation for surgical skills training, and establishing a didactic lecture series.

Results and Conclusions

We present a multimodal approach for using practical videoconferencing tools that provide the means for audiovisual communication to help augment residents' operative experience and limit the impact of self-isolation, redeployment, and limited operative exposure on surgical training.

OR black box and surgical control tower: Recording and streaming data and analytics to improve surgical care

Effective and safe surgery results from a complex sociotechnical process prone to human error. Acquiring large amount of data on surgical care and modelling the process of surgery with artificial intelligence's computational methods could shed lights on system strengths and limitations and enable computer-based smart assistance. With this vision in mind, surgeons and computer scientists have joined forces in a novel discipline called Surgical Data Science. In this regard, operating room (OR) black boxes and surgical control towers are being developed to systematically capture comprehensive data on surgical procedures and to oversee and assist during operating rooms activities, respectively. Most of the early Surgical Data Science works have focused on understanding risks and resilience factors affecting surgical safety, the context and workflow of procedures, and team behaviors. These pioneering efforts in sensing and analyzing surgical activities, together with the advent of precise robotic actuators, bring surgery on the verge of a fourth revolution characterized by smart assistance in perceptual, cognitive and physical tasks. Barriers to implement this vision exist, but the surgical-technical partnerships set by ambitious efforts such as the OR black box and the surgical control tower are working to overcome these roadblocks and translate the vision and early works described in the manuscript into value for patients, surgeons and health systems.

Artificial Intelligence for Intraoperative Guidance: Using Semantic Segmentation to Identify Surgical Anatomy During Laparoscopic Cholecystectomy

OBJECTIVE

To develop and evaluate the performance of artificial intelligence (AI) models that can identify safe and dangerous zones of dissection, and anatomical landmarks during laparoscopic cholecystectomy (LC).

SUMMARY BACKGROUND DATA

Many adverse events during surgery occur due to errors in visual perception and judgment leading to misinterpretation of anatomy. Deep learning, a subfield of AI, can potentially be used to provide real-time guidance intraoperatively.

METHODS

Deep learning models were developed and trained to identify safe (Go) and dangerous (No-Go) zones of dissection, liver, gallbladder, and hepatocystic triangle during LC. Annotations were performed by four high-volume surgeons. AI predictions were evaluated using 10-fold cross-validation against annotations by expert surgeons. Primary outcomes were intersection-over-union (IOU) and F1 score (validated spatial correlation indices), and secondary outcomes were pixel-wise accuracy, sensitivity, specificity, ± standard deviation.

RESULTS

AI models were trained on 2627 random frames from 290 LC videos, procured from 37 countries, 136 institutions and 153 surgeons. Mean IOU, F1 score, accuracy, sensitivity, and specificity for the AI to identify Go zones were 0.53 (±0.24), 0.70 (±0.28), 0.94 (±0.05), 0.69 (±0.20) and 0.94 (±0.03) respectively. For No-Go zones, these metrics were 0.71 (±0.29), 0.83 (±0.31), 0.95 (±0.06), 0.80 (±0.21) and 0.98 (±0.05), respectively. Mean IOU for identification of the liver, gallbladder and hepatocystic triangle were: 0.86 (±0.12), 0.72 (±0.19) and 0.65 (±0.22), respectively.

CONCLUSIONS

AI can be used to identify anatomy within the surgical field. This technology may eventually be used to provide real-time guidance and minimize the risk of adverse events.

The future of general surgery training: A Canadian resident nationwide Delphi consensus statement

BACKGROUND

Several models have been introduced to improve and restructure surgical training, but continued barriers exist. Residents are uniquely positioned to offer perspective on practical challenges and needs of reformatting surgical education. This study aimed to establish a nationwide, Delphi consensus statement on the perceptions of Canadian residents regarding the future of general surgery training.

METHODS

Canadian general surgery residents participated in a moderated focus group using the Nominal Group Technique to discuss early subspecialization, competency-based medical education, and transition to practice. Qualitative verbal data were transcribed, categorized into themes, and synthesized into recommendation statements. During an iterative Delphi survey, resident leaders ranked each statement on a 5-point Likert scale of agreement. The survey was terminated once consensus was achieved (≥2 survey rounds and Cronbach's α ≥ 0.80).

RESULTS

A total of 66 statements were synthesized by 16 members of the Canadian Association of General Surgeons Resident Committee. A total of 49 residents participated in the Delphi consensus, which was achieved after 2 voting rounds (Cronbach's α = 0.93). Participants agreed that (1) residency should focus on achieving standardized competencies and milestones based on resident ability to meet specific measurable metrics, (2) early streaming should be offered after "core" milestones and competencies have been achieved, and (3) an explicit period should allow transition-to-independent practice with tailored rotations, greater autonomy, and resident-run clinics. We identified 10 barriers to competency-based medical education implementation.

CONCLUSION

A nationwide consensus regarding the future of surgical training was established among current residents. These findings can inform and help implement guidelines and national curricula that meet the needs of the trainee and address the many challenges they face during their training.

Refining How We Define Laparoscopic Expertise

BACKGROUND

Traditional stratification of expertise in laparoscopic simulation assigns participants to novice, intermediate, or expert groups based on case numbers. We hypothesized that expert video assessment might refine this discrimination of psychomotor expertise, especially in light of new measurable parameters.

MATERIALS AND METHODS

One hundred five participants performed a defined intracorporeal suturing task in the pediatric laparoscopic surgery simulator armed with force-sensing capabilities. Participants were stratified into novice, intermediate, and expert groups via three classification schemes: (1) number of complex laparoscopic cases, (2) self-declared level of expertise, and (3) average expert rating of participants' videos. Precision, time to task completion, and force analysis parameters (FAP = total, maximum and mean forces in three axes) were compared using one-way analysis of variance tests. P < .05 was considered significant.

RESULTS

Participants stratified on the basis of case numbers and on the basis of self-declared level of expertise had statistically significant differences in time to task completion, but no significant difference in FAP. When participants were restratified according to expert assessment of their video performance, time to task completion as well as total and mean forces in X, Y, and Z axes allowed discrimination between novices, intermediates, and experts, thus establishing construct validity for the latter. Precision did not allow discrimination in any stratification scheme.

CONCLUSION

Compared with traditional stratification, video assessment allows refined discrimination of psychomotor expertise within a simulator. Assessment of FAP may become a relevant tool for teaching and assessing laparoscopic skills.

Measuring Decision-Making During Thyroidectomy: Validity Evidence for a Web-Based Assessment Tool

BACKGROUND

Errors in judgment during thyroidectomy can lead to recurrent laryngeal nerve injury and other complications. Despite the strong link between patient outcomes and intraoperative decision-making, methods to evaluate these complex skills are lacking. The purpose of this study was to develop objective metrics to evaluate advanced cognitive skills during thyroidectomy and to obtain validity evidence for them.

METHODS

An interactive online learning platform was developed ( www.thinklikeasurgeon.com ). Trainees and surgeons from four institutions completed a 33-item assessment, developed based on a cognitive task analysis and expert Delphi consensus. Sixteen items required subjects to make annotations on still frames of thyroidectomy videos, and accuracy scores were calculated based on an algorithm derived from experts' responses ("visual concordance test," VCT). Seven items were short answer (SA), requiring users to type their answers, and scores were automatically calculated based on their similarity to a pre-populated repertoire of correct responses. Test-retest reliability, internal consistency, and correlation of scores with self-reported experience and training level (novice, intermediate, expert) were calculated.

RESULTS

Twenty-eight subjects (10 endocrine surgeons and otolaryngologists, 18 trainees) participated. There was high test-retest reliability (intraclass correlation coefficient = 0.96; n = 10) and internal consistency (Cronbach's α = 0.93). The assessment demonstrated significant differences between novices, intermediates, and experts in total score (p < 0.01), VCT score (p < 0.01) and SA score (p < 0.01). There was high correlation between total case number and total score (ρ = 0.95, p < 0.01), between total case number and VCT score (ρ = 0.93, p < 0.01), and between total case number and SA score (ρ = 0.83, p < 0.01).

CONCLUSION

This study describes the development of novel metrics and provides validity evidence for an interactive Web-based platform to objectively assess decision-making during thyroidectomy.