1
|
Lund S, Navarro S, D'Angelo JD, Park YS, Rivera M. Expanded Access to Video-Based Laparoscopic Skills Assessments: Ease, Reliability, and Accuracy. JOURNAL OF SURGICAL EDUCATION 2024; 81:850-857. [PMID: 38664172 DOI: 10.1016/j.jsurg.2024.03.010] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/26/2023] [Revised: 01/29/2024] [Accepted: 03/13/2024] [Indexed: 05/13/2024]
Abstract
OBJECTIVE Video-based performance assessments provide essential feedback to surgical residents, but in-person and remote video-based assessment by trained proctors incurs significant cost. We aimed to determine the reliability, accuracy, and difficulty of untrained attending staff surgeon raters completing video-based assessments of a basic laparoscopic skill. Secondarily, we aimed to compare reliability and accuracy between 2 different types of assessment tools. DESIGN An anonymous survey was distributed electronically to surgical attendings via a national organizational listserv. Survey items included demographics, rating of video-based assessment experience (1 = have never completed video-based assessments, 5 = often complete video-based assessments), and rating of favorability toward video-based and in-person assessments (0 = not favorable, 100 = favorable). Participants watched 2 laparoscopic peg transfer performances, then rated each performance using an Objective Structured Assessment of Technical Skill (OSATS) form and the McGill Inanimate System for Training and Evaluation of Laparoscopic Skills (MISTELS). Participants then rated assessment completion ease (1 = Very Easy, 5 = Very Difficult). SETTING National survey of practicing surgeons. PARTICIPANTS Sixty-one surgery attendings with experience in laparoscopic surgery from 10 institutions participated as untrained raters. Six experienced laparoscopic skills proctors participated as expert raters. RESULTS Inter-rater reliability was substantial for both OSATS (k = 0.75) and MISTELS (k = 0.85). MISTELS accuracy was significantly higher than that of OSATS (κ: MISTELS = 0.18, 95%CI = [0.06,0.29]; OSATS = 0.02, 95%CI = [-0.01,0.04]). While participants were inexperienced with completing video-based assessments (median = 1/5), they perceived video-based assessments favorably (mean = 73.4) and felt assessment completion was "Easy" on average. CONCLUSIONS We demonstrate that faculty raters untrained in simulation-based assessments can successfully complete video-based assessments of basic laparoscopic skills with substantial inter-rater reliability without marked difficulty. These findings suggest an opportunity to increase access to feedback for trainees using video-based assessment of fundamental skills in laparoscopic surgery.
Collapse
Affiliation(s)
- Sarah Lund
- Mayo Clinic Department of Surgery, 200 1st Street SW, Rochester, Minnesota 55905.
| | - Sergio Navarro
- Mayo Clinic Department of Surgery, 200 1st Street SW, Rochester, Minnesota 55905
| | - Jonathan D D'Angelo
- Mayo Clinic Division of Colon and Rectal Surgery, 200 1st Street SW, Rochester, Minnesota 55905
| | - Yoon Soo Park
- Department of Medical Education, University of Illinois at Chicago College of Medicine, 808 S Wood Street, Chicago Illinois 60612
| | - Mariela Rivera
- Mayo Clinic Division of Trauma, Critical Care, and General Surgery, 200 1st Street SW, Rochester, Minnesota 55905
| |
Collapse
|
2
|
Lerner V, Arabkhazaeli M, DeStephano CC, Wu H, Chen CC. Modification of Fundamentals of Laparoscopic Surgery Manual Tasks Simulation Curriculum With the Addition of the Vaginal Cuff Closure Training. JOURNAL OF SURGICAL EDUCATION 2024; 81:122-133. [PMID: 38036386 DOI: 10.1016/j.jsurg.2023.10.013] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/26/2022] [Revised: 08/24/2023] [Accepted: 10/17/2023] [Indexed: 12/02/2023]
Abstract
OBJECTIVE Fundamentals of Laparoscopic Surgery (FLS) is a multiple-choice test and a manual skills exam using simulation that Obstetrics and Gynecology (OBGYN) residents must pass to qualify for board certification. There is insufficient validity evidence supporting the use of FLS as a high-stakes exam. This study examines the correlation between OBGYN residents' performance on the FLS manual tasks and simulated vaginal cuff closure. METHODS We compared residents' performance on FLS tasks with simulated vaginal cuff suturing on a model. During the first coached simulation session, after completion of training on the standard 5 FLS tasks, residents were coached on vaginal cuff closure using a simulated model placed inside the standard FLS box trainer. At a subsequent session, their performance was scored using the Global Operative Assessment of Laparoscopic Skills Scale (GOALS) and a second task-specific metric, and these scores were compared to their official FLS score. RESULTS Twenty-nine residents completed the vaginal cuff simulation training between June 2019 and November 2021. Nineteen of the 29 were able to complete the cuff closure with the mean time to completion being 14.5 minutes. We found no correlation between official manual skills FLS scores and vaginal cuff GOALS scores (rho = -0.02, p = 0.90) or cuff closure assessment tool score (rho = -0.015, p = 0.048). There was also no correlation between time to completion for any FLS task and vaginal cuff closure OSAT scores. All residents reported that they found the cuff to be a useful addition to the FLS curriculum. CONCLUSIONS Our study demonstrated that trainee performance on a simulated vaginal cuff closure model did not correlate with official FLS manual tasks skills. This finding adds to the body of evidence disputing the use of FLS as a high-stakes exam to assess laparoscopic skills in gynecology in the relationship with other variables category.
Collapse
Affiliation(s)
- Veronica Lerner
- Department of Obstetrics & Gynecology, Lenox Hill Hospital, Zucker School of Medicine at Hofstra/Northwell Health, New York, New York.
| | | | | | - Haotian Wu
- Department of Environmental Health Sciences, Columbia University Mailman School of Public Health, New York, New York
| | - Chi Chiung Chen
- Department of Gynecology and Obstetrics, Johns Hopkins University, Baltimore, Maryland
| |
Collapse
|
3
|
Higgins RM, Turbati MS, Goldblatt MI. Preparing for and passing the fundamentals of laparoscopic surgery (FLS) exam improves general surgery resident operative performance and autonomy. Surg Endosc 2023:10.1007/s00464-023-10124-8. [PMID: 37202525 DOI: 10.1007/s00464-023-10124-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/22/2023] [Accepted: 05/08/2023] [Indexed: 05/20/2023]
Abstract
BACKGROUND The American Board of Surgery made the Fundamentals of Laparoscopic Surgery (FLS) exam a prerequisite for board certification in 2009. Some residency programs have questioned the need for a continued FLS testing mandate given limited evidence that supports the impact of FLS on intraoperative skills. The Society for Improving Medical Professional Learning (SIMPL) app is a tool to evaluate resident intraoperative performance. We hypothesized that general surgery resident operative performance would improve immediately after preparing for the FLS exam. METHODS The national public FLS data registry was matched with SIMPL resident evaluations from 2015 to 2021 and de-identified. SIMPL evaluations are scored in three categories: supervision required (Zwisch scale 1-4, 1 = show and tell and 4 = supervision only), performance (scale 1-5, 1 = exceptional and 5 = unprepared), and case complexity (scale 1-3, 1 = easiest and 3 = hardest). Statistical analyses compared pre and post-FLS exam resident average operative evaluation scores. RESULTS There were a total of 76 general surgery residents, and 573 resident SIMPL evaluations included in this study. Residents required more supervision in laparoscopic cases performed before compared to after the FLS exam (2.84 vs. 3.03, respectively, p = 0.007). Residents performance scores improved from cases before compared to after the FLS exam (2.70 vs. 2.43, respectively, p = 0.001). Case complexity did not differ before versus after the FLS exam (2.13 vs. 2.18, respectively, p = 0.202). PGY level significantly predicted evaluation scores with a moderate correlation. A sub analysis grouped by PGY level revealed a significant improvement after the FLS exam in supervision among PGY-2 residents (2.33 vs. 2.58, respectively, p = 0.04) and performance among PGY-4 residents (2.67 vs 2.04, respectively, p < 0.001). CONCLUSIONS Preparation for, and passing, the FLS exam improves resident intraoperative laparoscopic performance and independence. We recommend taking the exam in the first two years of residency to enhance the laparoscopic experience for the remainder of training.
Collapse
Affiliation(s)
- Rana M Higgins
- Division of Minimally Invasive and Gastrointestinal Surgery, Department of Surgery, Medical College of Wisconsin, 8701 Watertown Plank Road, Milwaukee, WI, 53226, USA.
| | - Mia S Turbati
- Division of Minimally Invasive and Gastrointestinal Surgery, Department of Surgery, Medical College of Wisconsin, 8701 Watertown Plank Road, Milwaukee, WI, 53226, USA
| | - Matthew I Goldblatt
- Division of Minimally Invasive and Gastrointestinal Surgery, Department of Surgery, Medical College of Wisconsin, 8701 Watertown Plank Road, Milwaukee, WI, 53226, USA
| |
Collapse
|
4
|
Dedeilia A, Papapanou M, Papadopoulos AN, Karela NR, Androutsou A, Mitsopoulou D, Nikolakea M, Konstantinidis C, Papageorgakopoulou M, Sideris M, Johnson EO, Fitzpatrick S, Cometto G, Campbell J, Sotiropoulos MG. Health worker education during the COVID-19 pandemic: global disruption, responses and lessons for the future-a systematic review and meta-analysis. HUMAN RESOURCES FOR HEALTH 2023; 21:13. [PMID: 36829158 PMCID: PMC9951171 DOI: 10.1186/s12960-023-00799-4] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 03/22/2022] [Accepted: 02/01/2023] [Indexed: 06/18/2023]
Abstract
BACKGROUND This systematic review and meta-analysis identified early evidence quantifying the disruption to the education of health workers by the COVID-19 pandemic, ensuing policy responses and their outcomes. METHODS Following a pre-registered protocol and PRISMA/AMSTAR-2 guidelines, we systematically screened MEDLINE, EMBASE, Web of Science, CENTRAL, clinicaltrials.gov and Google Scholar from January 2020 to July 2022. We pooled proportion estimates via random-effects meta-analyses and explored subgroup differences by gender, occupational group, training stage, WHO regions/continents, and study end-year. We assessed risk of bias (Newcastle-Ottawa scale for observational studies, RοB2 for randomized controlled trials [RCT]) and rated evidence certainty using GRADE. RESULTS Of the 171 489 publications screened, 2 249 were eligible, incorporating 2 212 observational studies and 37 RCTs, representing feedback from 1 109 818 learners and 22 204 faculty. The sample mostly consisted of undergraduates, medical doctors, and studies from institutions in Asia. Perceived training disruption was estimated at 71.1% (95% confidence interval 67.9-74.2) and learner redeployment at 29.2% (25.3-33.2). About one in three learners screened positive for anxiety (32.3%, 28.5-36.2), depression (32.0%, 27.9-36.2), burnout (38.8%, 33.4-44.3) or insomnia (30.9%, 20.8-41.9). Policy responses included shifting to online learning, innovations in assessment, COVID-19-specific courses, volunteerism, and measures for learner safety. For outcomes of policy responses, most of the literature related to perceptions and preferences. More than two-thirds of learners (75.9%, 74.2-77.7) were satisfied with online learning (postgraduates more than undergraduates), while faculty satisfaction rate was slightly lower (71.8%, 66.7-76.7). Learners preferred an in-person component: blended learning 56.0% (51.2-60.7), face-to-face 48.8% (45.4-52.1), and online-only 32.0% (29.3-34.8). They supported continuation of the virtual format as part of a blended system (68.1%, 64.6-71.5). Subgroup differences provided valuable insights despite not resolving the considerable heterogeneity. All outcomes were assessed as very-low-certainty evidence. CONCLUSION The COVID-19 pandemic has severely disrupted health worker education, inflicting a substantial mental health burden on learners. Its impacts on career choices, volunteerism, pedagogical approaches and mental health of learners have implications for educational design, measures to protect and support learners, faculty and health workers, and workforce planning. Online learning may achieve learner satisfaction as part of a short-term solution or integrated into a blended model in the post-pandemic future.
Collapse
Affiliation(s)
- Aikaterini Dedeilia
- Department of Surgery, Massachusetts General Hospital, Boston, MA, USA
- Harvard Medical School, Boston, MA, USA
- HEALth Workforce Education (HEAL-Edu) Study Group, Athens, Greece
| | - Michail Papapanou
- HEALth Workforce Education (HEAL-Edu) Study Group, Athens, Greece
- School of Medicine, National and Kapodistrian University of Athens, Athens, Greece
- Society of Junior Doctors (SJD), Athens, Greece
| | - Andreas N Papadopoulos
- HEALth Workforce Education (HEAL-Edu) Study Group, Athens, Greece
- Department of Hygiene, Epidemiology and Medical Statistics, School of Medicine, National and Kapodistrian University of Athens, Athens, Greece
| | - Nina-Rafailia Karela
- HEALth Workforce Education (HEAL-Edu) Study Group, Athens, Greece
- School of Medicine, National and Kapodistrian University of Athens, Athens, Greece
| | - Anastasia Androutsou
- HEALth Workforce Education (HEAL-Edu) Study Group, Athens, Greece
- School of Medicine, National and Kapodistrian University of Athens, Athens, Greece
| | - Dimitra Mitsopoulou
- HEALth Workforce Education (HEAL-Edu) Study Group, Athens, Greece
- School of Medicine, National and Kapodistrian University of Athens, Athens, Greece
| | - Melina Nikolakea
- HEALth Workforce Education (HEAL-Edu) Study Group, Athens, Greece
- School of Medicine, National and Kapodistrian University of Athens, Athens, Greece
| | - Christos Konstantinidis
- HEALth Workforce Education (HEAL-Edu) Study Group, Athens, Greece
- School of Medicine, Aristotle University of Thessaloniki, Thessaloniki, Greece
| | - Manthia Papageorgakopoulou
- HEALth Workforce Education (HEAL-Edu) Study Group, Athens, Greece
- School of Medicine, University of Patras, Patras, Greece
| | - Michail Sideris
- Barts and the London School of Medicine and Dentistry, Queen Mary University of London, London, United Kingdom
| | | | | | - Giorgio Cometto
- Health Workforce Department, World Health Organization, Geneva, Switzerland
| | - Jim Campbell
- Health Workforce Department, World Health Organization, Geneva, Switzerland
| | - Marinos G Sotiropoulos
- Harvard Medical School, Boston, MA, USA.
- HEALth Workforce Education (HEAL-Edu) Study Group, Athens, Greece.
- Department of Neurology, Brigham and Women's Hospital & Massachusetts General Hospital, 55 Fruit Street, WACC721, Boston, MA, 02114, USA.
| |
Collapse
|