1
|
Yilmaz R, Fazlollahi AM, Winkler-Schwartz A, Wang A, Makhani HH, Alsayegh A, Bakhaidar M, Tran DH, Santaguida C, Del Maestro RF. Effect of Feedback Modality on Simulated Surgical Skills Learning Using Automated Educational Systems- A Four-Arm Randomized Control Trial. JOURNAL OF SURGICAL EDUCATION 2024; 81:275-287. [PMID: 38160107 DOI: 10.1016/j.jsurg.2023.11.001] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/03/2023] [Revised: 09/05/2023] [Accepted: 11/01/2023] [Indexed: 01/03/2024]
Abstract
OBJECTIVE To explore optimal feedback methodologies to enhance trainee skill acquisition in simulated surgical bimanual skills learning during brain tumor resections. HYPOTHESES (1) Providing feedback results in better learning outcomes in teaching surgical technical skill when compared to practice alone with no tailored performance feedback. (2) Providing more visual and visuospatial feedback results in better learning outcomes when compared to providing numerical feedback. DESIGN A prospective 4-parallel-arm randomized controlled trial. SETTING Neurosurgical Simulation and Artificial Intelligence Learning Centre, McGill University, Canada. PARTICIPANTS Medical students (n = 120) from 4 Quebec medical schools. RESULTS Participants completed a virtually simulated tumor resection task 5 times while receiving 1 of 4 feedback based on their group allocation: (1) practice-alone without feedback, (2) numerical feedback, (3) visual feedback, and (4) visuospatial feedback. Outcome measures were participants' scores on 14-performance metrics and the number of expert benchmarks achieved during each task. There were no significant differences in the first task which determined baseline performance. A statistically significant interaction between feedback allocation and task repetition was found on the number of benchmarks achieved, F (10.558, 408.257)=3.220, p < 0.001. Participants in all feedback groups significantly improved their performance compared to baseline. The visual feedback group achieved significantly higher number of benchmarks than the practice-alone group by the third repetition of the task, p = 0.005, 95%CI [0.42 3.25]. Visual feedback and visuospatial feedback improved performance significantly by the second repetition of the task, p = 0.016, 95%CI [0.19 2.71] and p = 0.003, 95%CI [0.4 2.57], respectively. CONCLUSION Simulations with autonomous visual computer assistance may be effective pedagogical tools in teaching bimanual operative skills via visual and visuospatial feedback information delivery.
Collapse
Affiliation(s)
- Recai Yilmaz
- Neurosurgical Simulation and Artificial Intelligence Learning Centre, Department of Neurology & Neurosurgery, Montreal Neurological Institute, McGill University, Montreal, Quebec, Canada; Faculty of Medicine and Health Sciences, McGill University, Montreal, Canada.
| | - Ali M Fazlollahi
- Neurosurgical Simulation and Artificial Intelligence Learning Centre, Department of Neurology & Neurosurgery, Montreal Neurological Institute, McGill University, Montreal, Quebec, Canada; Faculty of Medicine and Health Sciences, McGill University, Montreal, Canada
| | - Alexander Winkler-Schwartz
- Neurosurgical Simulation and Artificial Intelligence Learning Centre, Department of Neurology & Neurosurgery, Montreal Neurological Institute, McGill University, Montreal, Quebec, Canada; Department of Neurology and Neurosurgery, Montreal Neurological Institute and Hospital, McGill University, Montreal, Quebec, Canada
| | - Anna Wang
- Neurosurgical Simulation and Artificial Intelligence Learning Centre, Department of Neurology & Neurosurgery, Montreal Neurological Institute, McGill University, Montreal, Quebec, Canada
| | - Hafila Hassan Makhani
- Neurosurgical Simulation and Artificial Intelligence Learning Centre, Department of Neurology & Neurosurgery, Montreal Neurological Institute, McGill University, Montreal, Quebec, Canada
| | - Ahmad Alsayegh
- Neurosurgical Simulation and Artificial Intelligence Learning Centre, Department of Neurology & Neurosurgery, Montreal Neurological Institute, McGill University, Montreal, Quebec, Canada; Department of Neurology and Neurosurgery, Montreal Neurological Institute and Hospital, McGill University, Montreal, Quebec, Canada; Division of Neurosurgery, Department of Surgery, Faculty of Medicine, King Abdulaziz University, Jeddah, Saudi Arabia
| | - Mohamad Bakhaidar
- Neurosurgical Simulation and Artificial Intelligence Learning Centre, Department of Neurology & Neurosurgery, Montreal Neurological Institute, McGill University, Montreal, Quebec, Canada; Department of Neurology and Neurosurgery, Montreal Neurological Institute and Hospital, McGill University, Montreal, Quebec, Canada; Division of Neurosurgery, Department of Surgery, Faculty of Medicine, King Abdulaziz University, Jeddah, Saudi Arabia
| | - Dan Huy Tran
- Neurosurgical Simulation and Artificial Intelligence Learning Centre, Department of Neurology & Neurosurgery, Montreal Neurological Institute, McGill University, Montreal, Quebec, Canada
| | - Carlo Santaguida
- Department of Neurology and Neurosurgery, Montreal Neurological Institute and Hospital, McGill University, Montreal, Quebec, Canada
| | - Rolando F Del Maestro
- Neurosurgical Simulation and Artificial Intelligence Learning Centre, Department of Neurology & Neurosurgery, Montreal Neurological Institute, McGill University, Montreal, Quebec, Canada; Faculty of Medicine and Health Sciences, McGill University, Montreal, Canada; Department of Neurology and Neurosurgery, Montreal Neurological Institute and Hospital, McGill University, Montreal, Quebec, Canada
| |
Collapse
|
2
|
Almansouri A, Abou Hamdan N, Yilmaz R, Tee T, Pachchigar P, Eskandari M, Agu C, Giglio B, Balasubramaniam N, Bierbrier J, Collins DL, Gueziri HE, Del Maestro RF. Continuous Instrument Tracking in a Cerebral Corticectomy Ex Vivo Calf Brain Simulation Model: Face and Content Validation. Oper Neurosurg (Hagerstown) 2024:01787389-990000000-01017. [PMID: 38190098 DOI: 10.1227/ons.0000000000001044] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/17/2023] [Accepted: 11/13/2023] [Indexed: 01/09/2024] Open
Abstract
BACKGROUND AND OBJECTIVES Subpial corticectomy involving complete lesion resection while preserving pial membranes and avoiding injury to adjacent normal tissues is an essential bimanual task necessary for neurosurgical trainees to master. We sought to develop an ex vivo calf brain corticectomy simulation model with continuous assessment of surgical instrument movement during the simulation. A case series study of skilled participants was performed to assess face and content validity to gain insights into the utility of this training platform, along with determining if skilled and less skilled participants had statistical differences in validity assessment. METHODS An ex vivo calf brain simulation model was developed in which trainees performed a subpial corticectomy of three defined areas. A case series study assessed face and content validity of the model using 7-point Likert scale questionnaires. RESULTS Twelve skilled and 11 less skilled participants were included in this investigation. Overall median scores of 6.0 (range 4.0-6.0) for face validity and 6.0 (range 3.5-7.0) for content validity were determined on the 7-point Likert scale, with no statistical differences between skilled and less skilled groups identified. CONCLUSION A novel ex vivo calf brain simulator was developed to replicate the subpial resection procedure and demonstrated face and content validity.
Collapse
Affiliation(s)
- Abdulrahman Almansouri
- Neurosurgical Simulation and Artificial Intelligence Learning Centre, Department of Neurology and Neurosurgery, Montreal Neurological Institute, McGill University, Montreal, Quebec, Canada
- Department of Neurology and Neurosurgery, Montreal Neurological Institute and Hospital, McGill University, Montreal, Quebec, Canada
| | - Nour Abou Hamdan
- Neurosurgical Simulation and Artificial Intelligence Learning Centre, Department of Neurology and Neurosurgery, Montreal Neurological Institute, McGill University, Montreal, Quebec, Canada
| | - Recai Yilmaz
- Neurosurgical Simulation and Artificial Intelligence Learning Centre, Department of Neurology and Neurosurgery, Montreal Neurological Institute, McGill University, Montreal, Quebec, Canada
| | - Trisha Tee
- Neurosurgical Simulation and Artificial Intelligence Learning Centre, Department of Neurology and Neurosurgery, Montreal Neurological Institute, McGill University, Montreal, Quebec, Canada
| | - Puja Pachchigar
- Neurosurgical Simulation and Artificial Intelligence Learning Centre, Department of Neurology and Neurosurgery, Montreal Neurological Institute, McGill University, Montreal, Quebec, Canada
| | | | - Chinyelum Agu
- Neurosurgical Simulation and Artificial Intelligence Learning Centre, Department of Neurology and Neurosurgery, Montreal Neurological Institute, McGill University, Montreal, Quebec, Canada
| | - Bianca Giglio
- Neurosurgical Simulation and Artificial Intelligence Learning Centre, Department of Neurology and Neurosurgery, Montreal Neurological Institute, McGill University, Montreal, Quebec, Canada
| | - Neevya Balasubramaniam
- Neurosurgical Simulation and Artificial Intelligence Learning Centre, Department of Neurology and Neurosurgery, Montreal Neurological Institute, McGill University, Montreal, Quebec, Canada
| | - Joshua Bierbrier
- Department of Biomedical Engineering, McGill University, Montreal, Quebec, Canada
| | - D Louis Collins
- Department of Neurology and Neurosurgery, Montreal Neurological Institute and Hospital, McGill University, Montreal, Quebec, Canada
- Department of Biomedical Engineering, McGill University, Montreal, Quebec, Canada
| | - Houssem-Eddine Gueziri
- Department of Neurology and Neurosurgery, Montreal Neurological Institute and Hospital, McGill University, Montreal, Quebec, Canada
| | - Rolando F Del Maestro
- Neurosurgical Simulation and Artificial Intelligence Learning Centre, Department of Neurology and Neurosurgery, Montreal Neurological Institute, McGill University, Montreal, Quebec, Canada
- Department of Neurology and Neurosurgery, Montreal Neurological Institute and Hospital, McGill University, Montreal, Quebec, Canada
| |
Collapse
|
3
|
Fazlollahi AM, Yilmaz R, Winkler-Schwartz A, Mirchi N, Ledwos N, Bakhaidar M, Alsayegh A, Del Maestro RF. AI in Surgical Curriculum Design and Unintended Outcomes for Technical Competencies in Simulation Training. JAMA Netw Open 2023; 6:e2334658. [PMID: 37725373 PMCID: PMC10509729 DOI: 10.1001/jamanetworkopen.2023.34658] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 05/30/2023] [Accepted: 08/06/2023] [Indexed: 09/21/2023] Open
Abstract
Importance To better elucidate the role of artificial intelligence (AI) in surgical skills training requires investigations in the potential existence of a hidden curriculum. Objective To assess the pedagogical value of AI-selected technical competencies and their extended effects in surgical simulation training. Design, Setting, and Participants This cohort study was a follow-up of a randomized clinical trial conducted at the Neurosurgical Simulation and Artificial Intelligence Learning Centre at the Montreal Neurological Institute, McGill University, Montreal, Canada. Surgical performance metrics of medical students exposed to an AI-enhanced training curriculum were compared with a control group of participants who received no feedback and with expert benchmarks. Cross-sectional data were collected from January to April 2021 from medical students and from March 2015 to May 2016 from experts. This follow-up secondary analysis was conducted from June to September 2022. Participants included medical students (undergraduate year 0-2) in the intervention cohorts and neurosurgeons to establish expertise benchmarks. Exposure Performance assessment and personalized feedback by an intelligent tutor on 4 AI-selected learning objectives during simulation training. Main Outcomes and Measures Outcomes of interest were unintended performance outcomes, measured by significant within-participant difference from baseline in 270 performance metrics in the intervention cohort that was not observed in the control cohort. Results A total of 46 medical students (median [range] age, 22 [18-27] years; 27 [59%] women) and 14 surgeons (median [range] age, 45 [35-59] years; 14 [100%] men) were included in this study, and no participant was lost to follow-up. Feedback on 4 AI-selected technical competencies was associated with additional performance change in 32 metrics over the entire procedure and 20 metrics during tumor removal that was not observed in the control group. Participants exposed to the AI-enhanced curriculum demonstrated significant improvement in safety metrics, such as reducing the rate of healthy tissue removal (mean difference, -7.05 × 10-5 [95% CI, -1.09 × 10-4 to -3.14 × 10-5] mm3 per 20 ms; P < .001) and maintaining a focused bimanual control of the operative field (mean difference in maximum instrument divergence, -4.99 [95% CI, -8.48 to -1.49] mm, P = .006) compared with the control group. However, negative unintended effects were also observed. These included a significantly lower velocity and acceleration in the dominant hand (velocity: mean difference, -0.13 [95% CI, -0.17 to -0.09] mm per 20 ms; P < .001; acceleration: mean difference, -2.25 × 10-2 [95% CI, -3.20 × 10-2 to -1.31 × 10-2] mm per 20 ms2; P < .001) and a significant reduction in the rate of tumor removal (mean difference, -4.85 × 10-5 [95% CI, -7.22 × 10-5 to -2.48 × 10-5] mm3 per 20 ms; P < .001) compared with control. These unintended outcomes diverged students' movement and efficiency performance metrics away from the expertise benchmarks. Conclusions and Relevance In this cohort study of medical students, an AI-enhanced curriculum for bimanual surgical skills resulted in unintended changes that improved performance in safety but negatively affected some efficiency metrics. Incorporating AI in course design requires ongoing assessment to maintain transparency and foster evidence-based learning objectives.
Collapse
Affiliation(s)
- Ali M. Fazlollahi
- Neurosurgical Simulation and Artificial Intelligence Learning Centre, Department of Neurology and Neurosurgery, Montreal Neurological Institute and Hospital, McGill University, Montreal, Quebec, Canada
- Faculty of Medicine and Health Sciences, McGill University, Montreal, Quebec, Canada
| | - Recai Yilmaz
- Neurosurgical Simulation and Artificial Intelligence Learning Centre, Department of Neurology and Neurosurgery, Montreal Neurological Institute and Hospital, McGill University, Montreal, Quebec, Canada
- Faculty of Medicine and Health Sciences, McGill University, Montreal, Quebec, Canada
| | - Alexander Winkler-Schwartz
- Neurosurgical Simulation and Artificial Intelligence Learning Centre, Department of Neurology and Neurosurgery, Montreal Neurological Institute and Hospital, McGill University, Montreal, Quebec, Canada
| | - Nykan Mirchi
- Neurosurgical Simulation and Artificial Intelligence Learning Centre, Department of Neurology and Neurosurgery, Montreal Neurological Institute and Hospital, McGill University, Montreal, Quebec, Canada
| | - Nicole Ledwos
- Neurosurgical Simulation and Artificial Intelligence Learning Centre, Department of Neurology and Neurosurgery, Montreal Neurological Institute and Hospital, McGill University, Montreal, Quebec, Canada
| | - Mohamad Bakhaidar
- Neurosurgical Simulation and Artificial Intelligence Learning Centre, Department of Neurology and Neurosurgery, Montreal Neurological Institute and Hospital, McGill University, Montreal, Quebec, Canada
- Division of Neurosurgery, Department of Surgery, Faculty of Medicine, King Abdulaziz University, Jeddah, Saudi Arabia
| | - Ahmad Alsayegh
- Neurosurgical Simulation and Artificial Intelligence Learning Centre, Department of Neurology and Neurosurgery, Montreal Neurological Institute and Hospital, McGill University, Montreal, Quebec, Canada
- Division of Neurosurgery, Department of Surgery, Faculty of Medicine, King Abdulaziz University, Jeddah, Saudi Arabia
| | - Rolando F. Del Maestro
- Neurosurgical Simulation and Artificial Intelligence Learning Centre, Department of Neurology and Neurosurgery, Montreal Neurological Institute and Hospital, McGill University, Montreal, Quebec, Canada
- Faculty of Medicine and Health Sciences, McGill University, Montreal, Quebec, Canada
| |
Collapse
|
4
|
Titov O, Bykanov A, Pitskhelauri D. Neurosurgical skills analysis by machine learning models: systematic review. Neurosurg Rev 2023; 46:121. [PMID: 37191734 DOI: 10.1007/s10143-023-02028-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/26/2023] [Revised: 04/16/2023] [Accepted: 05/06/2023] [Indexed: 05/17/2023]
Abstract
Machine learning (ML) models are being actively used in modern medicine, including neurosurgery. This study aimed to summarize the current applications of ML in the analysis and assessment of neurosurgical skills. We conducted this systematic review in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. We searched the PubMed and Google Scholar databases for eligible studies published until November 15, 2022, and used the Medical Education Research Study Quality Instrument (MERSQI) to assess the quality of the included articles. Of the 261 studies identified, we included 17 in the final analysis. Studies were most commonly related to oncological, spinal, and vascular neurosurgery using microsurgical and endoscopic techniques. Machine learning-evaluated tasks included subpial brain tumor resection, anterior cervical discectomy and fusion, hemostasis of the lacerated internal carotid artery, brain vessel dissection and suturing, glove microsuturing, lumbar hemilaminectomy, and bone drilling. The data sources included files extracted from VR simulators and microscopic and endoscopic videos. The ML application was aimed at classifying participants into several expertise levels, analysis of differences between experts and novices, surgical instrument recognition, division of operation into phases, and prediction of blood loss. In two articles, ML models were compared with those of human experts. The machines outperformed humans in all tasks. The most popular algorithms used to classify surgeons by skill level were the support vector machine and k-nearest neighbors, and their accuracy exceeded 90%. The "you only look once" detector and RetinaNet usually solved the problem of detecting surgical instruments - their accuracy was approximately 70%. The experts differed by more confident contact with tissues, higher bimanuality, smaller distance between the instrument tips, and relaxed and focused state of the mind. The average MERSQI score was 13.9 (from 18). There is growing interest in the use of ML in neurosurgical training. Most studies have focused on the evaluation of microsurgical skills in oncological neurosurgery and on the use of virtual simulators; however, other subspecialties, skills, and simulators are being investigated. Machine learning models effectively solve different neurosurgical tasks related to skill classification, object detection, and outcome prediction. Properly trained ML models outperform human efficacy. Further research on ML application in neurosurgery is needed.
Collapse
Affiliation(s)
- Oleg Titov
- Burdenko Neurosurgery Center, Moscow, Russia.
- OPEN BRAIN, Laboratory of Neurosurgical Innovations, Moscow, Russia.
| | | | | |
Collapse
|
5
|
Park JJ, Tiefenbach J, Demetriades AK. The role of artificial intelligence in surgical simulation. FRONTIERS IN MEDICAL TECHNOLOGY 2022; 4:1076755. [PMID: 36590155 PMCID: PMC9794840 DOI: 10.3389/fmedt.2022.1076755] [Citation(s) in RCA: 16] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/21/2022] [Accepted: 11/21/2022] [Indexed: 12/15/2022] Open
Abstract
Artificial Intelligence (AI) plays an integral role in enhancing the quality of surgical simulation, which is increasingly becoming a popular tool for enriching the training experience of a surgeon. This spans the spectrum from facilitating preoperative planning, to intraoperative visualisation and guidance, ultimately with the aim of improving patient safety. Although arguably still in its early stages of widespread clinical application, AI technology enables personal evaluation and provides personalised feedback in surgical training simulations. Several forms of surgical visualisation technologies currently in use for anatomical education and presurgical assessment rely on different AI algorithms. However, while it is promising to see clinical examples and technological reports attesting to the efficacy of AI-supported surgical simulators, barriers to wide-spread commercialisation of such devices and software remain complex and multifactorial. High implementation and production costs, scarcity of reports evidencing the superiority of such technology, and intrinsic technological limitations remain at the forefront. As AI technology is key to driving the future of surgical simulation, this paper will review the literature delineating its current state, challenges, and prospects. In addition, a consolidated list of FDA/CE approved AI-powered medical devices for surgical simulation is presented, in order to shed light on the existing gap between academic achievements and the universal commercialisation of AI-enabled simulators. We call for further clinical assessment of AI-supported surgical simulators to support novel regulatory body approved devices and usher surgery into a new era of surgical education.
Collapse
Affiliation(s)
- Jay J. Park
- Department of General Surgery, Norfolk and Norwich University Hospital, Norwich, United Kingdom,Edinburgh Medical School, University of Edinburgh, Edinburgh, United Kingdom
| | - Jakov Tiefenbach
- Neurological Institute, Cleveland Clinic, Cleveland, OH, United States
| | - Andreas K. Demetriades
- Edinburgh Medical School, University of Edinburgh, Edinburgh, United Kingdom,Department of Neurosurgery, Royal Infirmary of Edinburgh, Edinburgh, United Kingdom
| |
Collapse
|
6
|
Chou D. Commentary: Artificial Neural Network Approach to Competency-Based Training Using a Virtual Reality Neurosurgical Simulation. Oper Neurosurg (Hagerstown) 2022; 23:e31-e32. [PMID: 35726933 DOI: 10.1227/ons.0000000000000176] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/20/2021] [Accepted: 12/24/2021] [Indexed: 11/18/2022] Open
Affiliation(s)
- Dean Chou
- Department of Neurosurgery, University of California San Francisco, San Francisco, California, USA
| |
Collapse
|