Effect of Artificial Intelligence Tutoring vs Expert Instruction on Learning Simulated Surgical Skills Among Medical Students: A Randomized Clinical Trial

IMPORTANCE

To better understand the emerging role of artificial intelligence (AI) in surgical training, efficacy of AI tutoring systems, such as the Virtual Operative Assistant (VOA), must be tested and compared with conventional approaches.

OBJECTIVE

To determine how VOA and remote expert instruction compare in learners' skill acquisition, affective, and cognitive outcomes during surgical simulation training.

DESIGN, SETTING, AND PARTICIPANTS

This instructor-blinded randomized clinical trial included medical students (undergraduate years 0-2) from 4 institutions in Canada during a single simulation training at McGill Neurosurgical Simulation and Artificial Intelligence Learning Centre, Montreal, Canada. Cross-sectional data were collected from January to April 2021. Analysis was conducted based on intention-to-treat. Data were analyzed from April to June 2021.

INTERVENTIONS

The interventions included 5 feedback sessions, 5 minutes each, during a single 75-minute training, including 5 practice sessions followed by 1 realistic virtual reality brain tumor resection. The 3 intervention arms included 2 treatment groups, AI audiovisual metric-based feedback (VOA group) and synchronous verbal scripted debriefing and instruction from a remote expert (instructor group), and a control group that received no feedback.

MAIN OUTCOMES AND MEASURES

The coprimary outcomes were change in procedural performance, quantified as Expertise Score by a validated assessment algorithm (Intelligent Continuous Expertise Monitoring System [ICEMS]; range, -1.00 to 1.00) for each practice resection, and learning and retention, measured from performance in realistic resections by ICEMS and blinded Objective Structured Assessment of Technical Skills (OSATS; range 1-7). Secondary outcomes included strength of emotions before, during, and after the intervention and cognitive load after intervention, measured in self-reports.

RESULTS

A total of 70 medical students (41 [59%] women and 29 [41%] men; mean [SD] age, 21.8 [2.3] years) from 4 institutions were randomized, including 23 students in the VOA group, 24 students in the instructor group, and 23 students in the control group. All participants were included in the final analysis. ICEMS assessed 350 practice resections, and ICEMS and OSATS evaluated 70 realistic resections. VOA significantly improved practice Expertise Scores by 0.66 (95% CI, 0.55 to 0.77) points compared with the instructor group and by 0.65 (95% CI, 0.54 to 0.77) points compared with the control group (P < .001). Realistic Expertise Scores were significantly higher for the VOA group compared with instructor (mean difference, 0.53 [95% CI, 0.40 to 0.67] points; P < .001) and control (mean difference. 0.49 [95% CI, 0.34 to 0.61] points; P < .001) groups. Mean global OSATS ratings were not statistically significant among the VOA (4.63 [95% CI, 4.06 to 5.20] points), instructor (4.40 [95% CI, 3.88-4.91] points), and control (3.86 [95% CI, 3.44 to 4.27] points) groups. However, on the OSATS subscores, VOA significantly enhanced the mean OSATS overall subscore compared with the control group (mean difference, 1.04 [95% CI, 0.13 to 1.96] points; P = .02), whereas expert instruction significantly improved OSATS subscores for instrument handling vs control (mean difference, 1.18 [95% CI, 0.22 to 2.14]; P = .01). No significant differences in cognitive load, positive activating, and negative emotions were found.

CONCLUSIONS AND RELEVANCE

In this randomized clinical trial, VOA feedback demonstrated superior performance outcome and skill transfer, with equivalent OSATS ratings and cognitive and emotional responses compared with remote expert instruction, indicating advantages for its use in simulation training.

TRIAL REGISTRATION

ClinicalTrials.gov Identifier: NCT04700384.

Continuous monitoring of surgical bimanual expertise using deep neural networks in virtual reality simulation

In procedural-based medicine, the technical ability can be a critical determinant of patient outcomes. Psychomotor performance occurs in real-time, hence a continuous assessment is necessary to provide action-oriented feedback and error avoidance guidance. We outline a deep learning application, the Intelligent Continuous Expertise Monitoring System (ICEMS), to assess surgical bimanual performance at 0.2-s intervals. A long-short term memory network was built using neurosurgeon and student performance in 156 virtually simulated tumor resection tasks. Algorithm predictive ability was tested separately on 144 procedures by scoring the performance of neurosurgical trainees who are at different training stages. The ICEMS successfully differentiated between neurosurgeons, senior trainees, junior trainees, and students. Trainee average performance score correlated with the year of training in neurosurgery. Furthermore, coaching and risk assessment for critical metrics were demonstrated. This work presents a comprehensive technical skill monitoring system with predictive validation throughout surgical residency training, with the ability to detect errors.

Assessment of learning curves on a simulated neurosurgical task using metrics selected by artificial intelligence

OBJECTIVE

Understanding the variation of learning curves of experts and trainees for a given surgical procedure is important in implementing formative learning paradigms to accelerate mastery. The study objectives were to use artificial intelligence (AI)-derived metrics to determine the learning curves of participants in 4 groups with different expertise levels who performed a series of identical virtual reality (VR) subpial resection tasks and to identify learning curve differences among the 4 groups.

METHODS

A total of 50 individuals participated, 14 neurosurgeons, 4 neurosurgical fellows and 10 senior residents (seniors), 10 junior residents (juniors), and 12 medical students. All participants performed 5 repetitions of a subpial tumor resection on the NeuroVR (CAE Healthcare) platform, and 6 a priori-derived metrics selected using the K-nearest neighbors machine learning algorithm were used to assess participant learning curves. Group learning curves were plotted over the 5 trials for each metric. A mixed, repeated-measures ANOVA was performed between the first and fifth trial. For significant interactions (p < 0.05), post hoc Tukey's HSD analysis was conducted to determine the location of the significance.

RESULTS

Overall, 5 of the 6 metrics assessed had a significant interaction (p < 0.05). The 4 groups, neurosurgeons, seniors, juniors, and medical students, showed an improvement between the first and fifth trial on at least one of the 6 metrics evaluated.

CONCLUSIONS

Learning curves generated using AI-derived metrics provided novel insights into technical skill acquisition, based on expertise level, during repeated VR-simulated subpial tumor resections, which will allow educators to develop more focused formative educational paradigms for neurosurgical trainees.

Essential oil compositions of Cymbopogon parkeri STAPF from Iran

Aerial parts of aromatic grass, Cymbopogon parkeri STAPF, were collected at flowering stage from Kerman province of Iran. The essential oil of air dried samples obtained by hydro-distillation method. The compositions of the essential oil were determined by the use of GC and GC-MS. Nineteen (98.7%) constituents were identified. The main constituents were piperitone (80.8%), germacrene-D (5.1%), santolinyl acetate (2.1%) and alpha-eudesmol (2.1%).

Nondominant Hand Skills Spatial and Psychomotor Analysis During a Complex Virtual Reality Neurosurgical Task-A Case Series Study

BACKGROUND

Virtual reality surgical simulators provide detailed psychomotor performance data, allowing qualitative and quantitative assessment of hand function. The nondominant hand plays an essential role in neurosurgery in exposing the operative area, assisting the dominant hand to optimize task execution, and hemostasis. Outlining expert-level nondominant hand skills may be critical to understand surgical expertise and aid learner training.

OBJECTIVE

To (1) provide validity for the simulated bimanual subpial tumor resection task and (2) to use this simulation in qualitative and quantitative evaluation of nondominant hand skills for bipolar forceps utilization.

METHODS

In this case series study, 45 right-handed participants performed a simulated subpial tumor resection using simulated bipolar forceps in the nondominant hand for assisting the surgery and hemostasis. A 10-item questionnaire was used to assess task validity. The nondominant hand skills across 4 expertise levels (neurosurgeons, senior trainees, junior trainees, and medical students) were analyzed by 2 visual models and performance metrics.

RESULTS

Neurosurgeon median (range) overall satisfaction with the simulated scenario was 4.0/5.0 (2.0-5.0). The visual models demonstrated a decrease in high force application areas on pial surface with increased expertise level. Bipolar-pia mater interactions were more focused around the tumoral region for neurosurgeons and senior trainees. These groups spent more time using the bipolar while interacting with pia. All groups spent significantly higher time in the left upper pial quadrant than other quadrants.

CONCLUSION

This work introduces new approaches for the evaluation of nondominant hand skills which may help surgical trainees by providing both qualitative and quantitative feedback.

Performance of an optimum receiver designed for pattern recognition with nonoverlapping target and scene noise

The design of an optimum receiver for pattern recognition is based on multiple-alternative hypothesis testing with unknown parameters for detecting and locating a noisy target or a noise-free target in scene noise that is spatially nonoverlapping with this target. The optimum receiver designed for a noise-free target has the interesting property of detecting, without error, a noise-free target that has unknown illumination by using operations that are independent of the scene-noise statistics. We investigate the performance of the optimum receiver designed for nonoverlapping target and scene noise in terms of rotation and scale sensitivity of the input targets and discrimination against similar objects. Because it is not possible in practical systems to have a completely noise-free target, we examine how the performance of the optimum receiver designed for a noise-free target is affected when there is some overlapping noise on the target. The application of the optimum receiver to binary character recognition is described. Computer simulation results are provided.

COMPARING THE EFFICIENCY OF A REAL-TIME ARTIFICIAL INTELLIGENCE INSTRUCTOR TO HUMAN EXPERT INSTRUCTORS IN SIMULATED SURGICAL TECHNICAL SKILLS TRAINING– A RANDOMIZED CONTROLLED TRIAL

BTFC travel award recipient

Artificial intelligence systems provide risk-free training on realistically simulated patient cases and objective assessment of surgical technical skills. This randomized controlled study compared a real-time intelligent tutoring system in technical skills learning with human expert instructor-mediated training.

METHODS: Ninety-eight medical students performed six simulated brain tumor resections. Participants were randomly allocated into (1)no-real-time feedback, (2)real-time intelligent instruction, and (3)in- person human instruction. All students performed the first repetition without receiving feedback (baseline). Group-1 received visual feedback only after each procedure based on expert benchmarks. Group-2 was instructed by the intelligent system in real-time. After each task, the students were shown their error-video clips generated by this system alongside expert-level demonstrations on how to improve. Group-3 was instructed by human instructors during the tasks. After each task, instructors summarized the areas of improvement and demonstrated correction techniques. Participant performance was scored by the intelligent system and also by blinded experts using OSATS scores. The performance score was compared within groups and between groups to compare learning. RESULTS: Compared to baseline performance, Group-2 and Group-3 significantly improved in the performance score by the third and second repetition, respectively (p<0.01, p=0.01). The between-groups comparison demonstrated that Group-2 scored significantly higher than Group-3 in the fifth repetition (p<0.01). Group-2 achieved significantly higher OSATS scores than Group-1 in the sixth task. CONCLUSIONS: Artificial intelligence may facilitate trainee learning by providing equally or more efficient learning when compared to human instruction. These systems may aid in developing competency-based standardized curricula in surgical training.

Enhancing trauma care through innovative trauma and disaster team response training: A blended learning approach in Tanzania

BACKGROUND

In Tanzania, inadequate infrastructures and shortages of trauma-response training exacerbate trauma-related fatalities. McGill University's Centre for Global Surgery introduced the Trauma and Disaster Team Response course (TDTR) to address these challenges. This study assesses the impact of simulation-based TDTR training on care providers' knowledge/skills and healthcare processes to enhance patient outcomes.

METHODS

The study used a pre-post-interventional design. TDTR, led by Tanzanian instructors at Muhimbili Orthopedic Institute from August 16-18, 2023, involved 22 participants in blended online and in-person approaches with simulated skills sessions. Validated tools assessed participants' knowledge/skills and teamwork pre/post-interventions, alongside feedback surveys. Outcome measures included evaluating 24-h emergency department patient arrival-to-care time pre-/post-TDTR interventions, analyzed using parametric and non-parametric tests based on data distributions.

RESULTS

Participants' self-assessment skills significantly improved (median increase from 34 to 58, p < 0.001), along with teamwork (median increase from 44.5 to 87.5, p < 0.003). While 99% of participants expressed satisfaction with TDTR meeting their expectations, 97% were interested in teaching future sessions. The six-month post-intervention arrival-to-care time significantly decreased from 29 to 13 min, indicating a 55.17% improvement (p < 0.004). The intervention led to fewer ward admissions (35.26% from 51.67%) and more directed to operating theaters (29.83% from 16.85%), suggesting improved patient management (p < 0.018).

CONCLUSION

The study confirmed surgical skills training effectiveness in Tanzanian settings, highlighting TDTR's role in improving teamwork and healthcare processes that enhanced patient outcomes. To sustain progress and empower independent trauma educators, ongoing refresher sessions and expanding TDTR across low- and middle-income countries are recommended to align with global surgery goals.

Effect of Feedback Modality on Simulated Surgical Skills Learning Using Automated Educational Systems- A Four-Arm Randomized Control Trial

OBJECTIVE

To explore optimal feedback methodologies to enhance trainee skill acquisition in simulated surgical bimanual skills learning during brain tumor resections.

HYPOTHESES

(1) Providing feedback results in better learning outcomes in teaching surgical technical skill when compared to practice alone with no tailored performance feedback. (2) Providing more visual and visuospatial feedback results in better learning outcomes when compared to providing numerical feedback.

DESIGN

A prospective 4-parallel-arm randomized controlled trial.

SETTING

Neurosurgical Simulation and Artificial Intelligence Learning Centre, McGill University, Canada.

PARTICIPANTS

Medical students (n = 120) from 4 Quebec medical schools.

RESULTS

Participants completed a virtually simulated tumor resection task 5 times while receiving 1 of 4 feedback based on their group allocation: (1) practice-alone without feedback, (2) numerical feedback, (3) visual feedback, and (4) visuospatial feedback. Outcome measures were participants' scores on 14-performance metrics and the number of expert benchmarks achieved during each task. There were no significant differences in the first task which determined baseline performance. A statistically significant interaction between feedback allocation and task repetition was found on the number of benchmarks achieved, F (10.558, 408.257)=3.220, p < 0.001. Participants in all feedback groups significantly improved their performance compared to baseline. The visual feedback group achieved significantly higher number of benchmarks than the practice-alone group by the third repetition of the task, p = 0.005, 95%CI [0.42 3.25]. Visual feedback and visuospatial feedback improved performance significantly by the second repetition of the task, p = 0.016, 95%CI [0.19 2.71] and p = 0.003, 95%CI [0.4 2.57], respectively.

CONCLUSION

Simulations with autonomous visual computer assistance may be effective pedagogical tools in teaching bimanual operative skills via visual and visuospatial feedback information delivery.

Real-Time multifaceted artificial intelligence vs In-Person instruction in teaching surgical technical skills: a randomized controlled trial

Trainees develop surgical technical skills by learning from experts who provide context for successful task completion, identify potential risks, and guide correct instrument handling. This expert-guided training faces significant limitations in objectively assessing skills in real-time and tracking learning. It is unknown whether AI systems can effectively replicate nuanced real-time feedback, risk identification, and guidance in mastering surgical technical skills that expert instructors offer. This randomized controlled trial compared real-time AI feedback to in-person expert instruction. Ninety-seven medical trainees completed a 90-min simulation training with five practice tumor resections followed by a realistic brain tumor resection. They were randomly assigned into 1-real-time AI feedback, 2-in-person expert instruction, and 3-no real-time feedback. Performance was assessed using a composite-score and Objective Structured Assessment of Technical Skills rating, rated by blinded experts. Training with real-time AI feedback (n = 33) resulted in significantly better performance outcomes compared to no real-time feedback (n = 32) and in-person instruction (n = 32), .266, [95% CI .107 .425], p < .001; .332, [95% CI .173 .491], p = .005, respectively. Learning from AI resulted in similar OSATS ratings (4.30 vs 4.11, p = 1) compared to in-person training with expert instruction. Intelligent systems may refine the way operating skills are taught, providing tailored, quantifiable feedback and actionable instructions in real-time.

Performance in a Simulated Virtual Reality Anterior Cervical Discectomy and Fusion Task: Disc Residual, Rate of Removal, and Efficiency Analyses

BACKGROUND AND OBJECTIVES

Anterior cervical discectomy and fusion (ACDF) is among the most common spine procedures. The Sim-Ortho virtual reality simulator platform contains a validated ACDF simulated task for performance assessment. This study aims to develop a methodology to extract three-dimensional data and reconstruct and quantitate specific simulated disc tissues to generate novel metrics to analyze performance metrics of skilled and less skilled participants.

METHODS

We used open-source platforms to develop a methodology to extract three-dimensional information from ACDF simulation data. Metrics generated included, efficiency index, disc volumes removed from defined regions, and rate of tissue removal from superficial, central, and deep disc regions. A pilot study was performed to assess the utility of this methodology to assess expertise during the ACDF simulated procedure.

RESULTS

The system outlined, extracts data allowing the development of a methodology which accurately reconstructs and quantitates 3-dimensional disc volumes. In the pilot study, data sets from 27 participants, divided into postresident, resident, and medical student groups, allowed assessment of multiple novel metrics, including efficiency index (surgical time spent in actively removing disc), where the postresident group spent 61.8% of their time compared with 53% and 30.2% for the resident and medical student groups, respectively ( P = .01). During the annulotomy component, the postresident group removed 47.4% more disc than the resident groups and 102% more than the medical student groups ( P = .03).

CONCLUSION

The methodology developed in this study generates novel surgical procedural metrics from 3-dimensional data generated by virtual reality simulators and can be used to assess surgical performance.

AI in Surgical Curriculum Design and Unintended Outcomes for Technical Competencies in Simulation Training

Importance

To better elucidate the role of artificial intelligence (AI) in surgical skills training requires investigations in the potential existence of a hidden curriculum.

Objective

To assess the pedagogical value of AI-selected technical competencies and their extended effects in surgical simulation training.

Design, Setting, and Participants

This cohort study was a follow-up of a randomized clinical trial conducted at the Neurosurgical Simulation and Artificial Intelligence Learning Centre at the Montreal Neurological Institute, McGill University, Montreal, Canada. Surgical performance metrics of medical students exposed to an AI-enhanced training curriculum were compared with a control group of participants who received no feedback and with expert benchmarks. Cross-sectional data were collected from January to April 2021 from medical students and from March 2015 to May 2016 from experts. This follow-up secondary analysis was conducted from June to September 2022. Participants included medical students (undergraduate year 0-2) in the intervention cohorts and neurosurgeons to establish expertise benchmarks.

Exposure

Performance assessment and personalized feedback by an intelligent tutor on 4 AI-selected learning objectives during simulation training.

Main Outcomes and Measures

Outcomes of interest were unintended performance outcomes, measured by significant within-participant difference from baseline in 270 performance metrics in the intervention cohort that was not observed in the control cohort.

Results

A total of 46 medical students (median [range] age, 22 [18-27] years; 27 [59%] women) and 14 surgeons (median [range] age, 45 [35-59] years; 14 [100%] men) were included in this study, and no participant was lost to follow-up. Feedback on 4 AI-selected technical competencies was associated with additional performance change in 32 metrics over the entire procedure and 20 metrics during tumor removal that was not observed in the control group. Participants exposed to the AI-enhanced curriculum demonstrated significant improvement in safety metrics, such as reducing the rate of healthy tissue removal (mean difference, -7.05 × 10-5 [95% CI, -1.09 × 10-4 to -3.14 × 10-5] mm3 per 20 ms; P < .001) and maintaining a focused bimanual control of the operative field (mean difference in maximum instrument divergence, -4.99 [95% CI, -8.48 to -1.49] mm, P = .006) compared with the control group. However, negative unintended effects were also observed. These included a significantly lower velocity and acceleration in the dominant hand (velocity: mean difference, -0.13 [95% CI, -0.17 to -0.09] mm per 20 ms; P < .001; acceleration: mean difference, -2.25 × 10-2 [95% CI, -3.20 × 10-2 to -1.31 × 10-2] mm per 20 ms2; P < .001) and a significant reduction in the rate of tumor removal (mean difference, -4.85 × 10-5 [95% CI, -7.22 × 10-5 to -2.48 × 10-5] mm3 per 20 ms; P < .001) compared with control. These unintended outcomes diverged students' movement and efficiency performance metrics away from the expertise benchmarks.

Conclusions and Relevance

In this cohort study of medical students, an AI-enhanced curriculum for bimanual surgical skills resulted in unintended changes that improved performance in safety but negatively affected some efficiency metrics. Incorporating AI in course design requires ongoing assessment to maintain transparency and foster evidence-based learning objectives.

ChatGPT and neurosurgical education: A crossroads of innovation and opportunity

Large language models (LLM) have been promising recently in the medical field, with numerous applications in clinical neuroscience. OpenAI's launch of Generative Pre-trained Transformer 3.5 (GPT-3.5) in November 2022 and its successor, Generative Pre-trained Transformer 4 (GPT 4) in March 2023 have garnered widespread attention and debate surrounding natural language processing (NLP) and LLM advancements. Transformer models are trained on natural language datasets to predict and generate sequences of characters. Using internal weights from training, they produce tokens that align with their understanding of the initial input. This paper delves into ChatGPT's potential as a learning tool in neurosurgery while contextualizing its abilities for passing medical licensing exams and neurosurgery written boards. Additionally, possibilities for creating personalized case presentations and study material are discussed alongside ChatGPT's capacity to optimize the research workflow and perform a concise literature review. However, such tools need to be used with caution, given the possibility of artificial intelligence hallucinations and other concerns such as user overreliance, and complacency. Overall, this opinion paper raises key points surrounding ChatGPT's role in neurosurgical education.