Learning Fundamentals of Laparoscopic Surgery Manual Skills: An Institutional Experience With Remote Coaching and Assessment

Expanded Access to Video-Based Laparoscopic Skills Assessments: Ease, Reliability, and Accuracy

OBJECTIVE

Video-based performance assessments provide essential feedback to surgical residents, but in-person and remote video-based assessment by trained proctors incurs significant cost. We aimed to determine the reliability, accuracy, and difficulty of untrained attending staff surgeon raters completing video-based assessments of a basic laparoscopic skill. Secondarily, we aimed to compare reliability and accuracy between 2 different types of assessment tools.

DESIGN

An anonymous survey was distributed electronically to surgical attendings via a national organizational listserv. Survey items included demographics, rating of video-based assessment experience (1 = have never completed video-based assessments, 5 = often complete video-based assessments), and rating of favorability toward video-based and in-person assessments (0 = not favorable, 100 = favorable). Participants watched 2 laparoscopic peg transfer performances, then rated each performance using an Objective Structured Assessment of Technical Skill (OSATS) form and the McGill Inanimate System for Training and Evaluation of Laparoscopic Skills (MISTELS). Participants then rated assessment completion ease (1 = Very Easy, 5 = Very Difficult).

SETTING

National survey of practicing surgeons.

PARTICIPANTS

Sixty-one surgery attendings with experience in laparoscopic surgery from 10 institutions participated as untrained raters. Six experienced laparoscopic skills proctors participated as expert raters.

RESULTS

Inter-rater reliability was substantial for both OSATS (k = 0.75) and MISTELS (k = 0.85). MISTELS accuracy was significantly higher than that of OSATS (κ: MISTELS = 0.18, 95%CI = [0.06,0.29]; OSATS = 0.02, 95%CI = [-0.01,0.04]). While participants were inexperienced with completing video-based assessments (median = 1/5), they perceived video-based assessments favorably (mean = 73.4) and felt assessment completion was "Easy" on average.

CONCLUSIONS

We demonstrate that faculty raters untrained in simulation-based assessments can successfully complete video-based assessments of basic laparoscopic skills with substantial inter-rater reliability without marked difficulty. These findings suggest an opportunity to increase access to feedback for trainees using video-based assessment of fundamental skills in laparoscopic surgery.

Modification of Fundamentals of Laparoscopic Surgery Manual Tasks Simulation Curriculum With the Addition of the Vaginal Cuff Closure Training

OBJECTIVE

Fundamentals of Laparoscopic Surgery (FLS) is a multiple-choice test and a manual skills exam using simulation that Obstetrics and Gynecology (OBGYN) residents must pass to qualify for board certification. There is insufficient validity evidence supporting the use of FLS as a high-stakes exam. This study examines the correlation between OBGYN residents' performance on the FLS manual tasks and simulated vaginal cuff closure.

METHODS

We compared residents' performance on FLS tasks with simulated vaginal cuff suturing on a model. During the first coached simulation session, after completion of training on the standard 5 FLS tasks, residents were coached on vaginal cuff closure using a simulated model placed inside the standard FLS box trainer. At a subsequent session, their performance was scored using the Global Operative Assessment of Laparoscopic Skills Scale (GOALS) and a second task-specific metric, and these scores were compared to their official FLS score.

RESULTS

Twenty-nine residents completed the vaginal cuff simulation training between June 2019 and November 2021. Nineteen of the 29 were able to complete the cuff closure with the mean time to completion being 14.5 minutes. We found no correlation between official manual skills FLS scores and vaginal cuff GOALS scores (rho = -0.02, p = 0.90) or cuff closure assessment tool score (rho = -0.015, p = 0.048). There was also no correlation between time to completion for any FLS task and vaginal cuff closure OSAT scores. All residents reported that they found the cuff to be a useful addition to the FLS curriculum.

CONCLUSIONS

Our study demonstrated that trainee performance on a simulated vaginal cuff closure model did not correlate with official FLS manual tasks skills. This finding adds to the body of evidence disputing the use of FLS as a high-stakes exam to assess laparoscopic skills in gynecology in the relationship with other variables category.

Preparing for and passing the fundamentals of laparoscopic surgery (FLS) exam improves general surgery resident operative performance and autonomy

BACKGROUND

The American Board of Surgery made the Fundamentals of Laparoscopic Surgery (FLS) exam a prerequisite for board certification in 2009. Some residency programs have questioned the need for a continued FLS testing mandate given limited evidence that supports the impact of FLS on intraoperative skills. The Society for Improving Medical Professional Learning (SIMPL) app is a tool to evaluate resident intraoperative performance. We hypothesized that general surgery resident operative performance would improve immediately after preparing for the FLS exam.

METHODS

The national public FLS data registry was matched with SIMPL resident evaluations from 2015 to 2021 and de-identified. SIMPL evaluations are scored in three categories: supervision required (Zwisch scale 1-4, 1 = show and tell and 4 = supervision only), performance (scale 1-5, 1 = exceptional and 5 = unprepared), and case complexity (scale 1-3, 1 = easiest and 3 = hardest). Statistical analyses compared pre and post-FLS exam resident average operative evaluation scores.

RESULTS

There were a total of 76 general surgery residents, and 573 resident SIMPL evaluations included in this study. Residents required more supervision in laparoscopic cases performed before compared to after the FLS exam (2.84 vs. 3.03, respectively, p = 0.007). Residents performance scores improved from cases before compared to after the FLS exam (2.70 vs. 2.43, respectively, p = 0.001). Case complexity did not differ before versus after the FLS exam (2.13 vs. 2.18, respectively, p = 0.202). PGY level significantly predicted evaluation scores with a moderate correlation. A sub analysis grouped by PGY level revealed a significant improvement after the FLS exam in supervision among PGY-2 residents (2.33 vs. 2.58, respectively, p = 0.04) and performance among PGY-4 residents (2.67 vs 2.04, respectively, p < 0.001).

CONCLUSIONS

Preparation for, and passing, the FLS exam improves resident intraoperative laparoscopic performance and independence. We recommend taking the exam in the first two years of residency to enhance the laparoscopic experience for the remainder of training.

Health worker education during the COVID-19 pandemic: global disruption, responses and lessons for the future-a systematic review and meta-analysis

BACKGROUND

This systematic review and meta-analysis identified early evidence quantifying the disruption to the education of health workers by the COVID-19 pandemic, ensuing policy responses and their outcomes.

METHODS

Following a pre-registered protocol and PRISMA/AMSTAR-2 guidelines, we systematically screened MEDLINE, EMBASE, Web of Science, CENTRAL, clinicaltrials.gov and Google Scholar from January 2020 to July 2022. We pooled proportion estimates via random-effects meta-analyses and explored subgroup differences by gender, occupational group, training stage, WHO regions/continents, and study end-year. We assessed risk of bias (Newcastle-Ottawa scale for observational studies, RοB2 for randomized controlled trials [RCT]) and rated evidence certainty using GRADE.

RESULTS

Of the 171 489 publications screened, 2 249 were eligible, incorporating 2 212 observational studies and 37 RCTs, representing feedback from 1 109 818 learners and 22 204 faculty. The sample mostly consisted of undergraduates, medical doctors, and studies from institutions in Asia. Perceived training disruption was estimated at 71.1% (95% confidence interval 67.9-74.2) and learner redeployment at 29.2% (25.3-33.2). About one in three learners screened positive for anxiety (32.3%, 28.5-36.2), depression (32.0%, 27.9-36.2), burnout (38.8%, 33.4-44.3) or insomnia (30.9%, 20.8-41.9). Policy responses included shifting to online learning, innovations in assessment, COVID-19-specific courses, volunteerism, and measures for learner safety. For outcomes of policy responses, most of the literature related to perceptions and preferences. More than two-thirds of learners (75.9%, 74.2-77.7) were satisfied with online learning (postgraduates more than undergraduates), while faculty satisfaction rate was slightly lower (71.8%, 66.7-76.7). Learners preferred an in-person component: blended learning 56.0% (51.2-60.7), face-to-face 48.8% (45.4-52.1), and online-only 32.0% (29.3-34.8). They supported continuation of the virtual format as part of a blended system (68.1%, 64.6-71.5). Subgroup differences provided valuable insights despite not resolving the considerable heterogeneity. All outcomes were assessed as very-low-certainty evidence.

CONCLUSION

The COVID-19 pandemic has severely disrupted health worker education, inflicting a substantial mental health burden on learners. Its impacts on career choices, volunteerism, pedagogical approaches and mental health of learners have implications for educational design, measures to protect and support learners, faculty and health workers, and workforce planning. Online learning may achieve learner satisfaction as part of a short-term solution or integrated into a blended model in the post-pandemic future.