Self-Assessment of Surgical Skills: A Systematic Review

Self-assessment, and not continuous training, improves basic open suturing skills

BACKGROUND

To develop and maintain suturing skills, clinical exposure is important. When clinical exposure cannot be guaranteed, an adequate training schedule for suturing skills is required. This study evaluates the effect of continuous training, 'reflection before practice' and self-assessment on basic open suturing skills.

METHODS

Medical students performed four basic suturing tasks on a simulation set up before ('pre-test') and after their surgical rotation ('after-test'). Participants were divided in three groups; the 'clinical exposure group' (n = 44) had clinical exposure during their rotation only, the 'continuous training group' (n = 16) completed a suturing interval training during their rotation and the 'self-assessment group' (n = 16) also completed a suturing interval training, but with the use of reflection before practice and self-assessment. Parameters measured by a tracking system during the suturing tasks and a calculated 'composite score' were compared between groups and test-moments.

RESULTS

A significantly better composite score was found at the after-test compared to the pre-test for all groups for all basic suturing tasks (0.001 ≤ p ≤ 0.049). The self-assessment group scored better at the pre-test than the other two groups for all tasks, except for 'knot tying by hand' (0.004 ≤ p ≤ 0.063). However, this group did not score better at the after-test for all tasks, compared to the other two groups. This resulted in a smaller delta of time ('transcutaneous suture', p = 0.013), distance ('Donati suture' and 'intracutaneous suture', 0.005 ≤ p ≤ 0.009) or composite score (all tasks, except for knot tying by hand, 0.007 ≤ p ≤ 0.061) in the self-assessment group.

CONCLUSION

Reflection before practice and self-assessment during continuous training of basic open suturing tasks, may improve surgical skills at the start of the learning curve.

Assessing competency in exodontia: A UK perspective

INTRODUCTION

Assessment ensures that undergraduate students and postgraduate trainees are providing optimal patient care. Numerous assessment methods are currently being used within the United Kingdom. Therefore, the purpose of this review was to provide an overview of the methods used to assess competency in exodontia and compare methods against criteria defined for good assessment.

MATERIALS AND METHODS

A systematic narrative review was carried out. Databases were electronically searched for published studies assessing the competence of undergraduate students and postgraduate trainees in exodontia up to October 2023.The validity, reproducibility, equivalence, feasibility, educational impact, catalytic effect and acceptability of each assessment method was considered.

RESULTS

In this review, 591 undergraduate students (n = 529) and postgraduate dental trainees (n = 62) had their competence in exodontia assessed. Undergraduate students were assessed on simple exodontia with forceps and elevators, whereas postgraduate trainees were assessed on the surgical removal of mandibular third molars. Only one study took place in a simulated dental environment, the remaining (n = 7) took place in a clinical environment. A range of formative and summative assessment methods were identified including structured clinical objective tests (SCOT), self-assessment, peer assessment and logbooks. Common marking criteria such as global rating and checklist scales were also highlighted.

CONCLUSION

This review highlights the lack of evidence to support one form of assessment and the lack of consensus on assessment within the United Kingdom. Further research related to this topic, alongside the opinion of trainees and trainers is required to improve assessments. This will maximise learning opportunities for trainees and translate to better patient care.

The Importance of Multiorgan Procurement in the Improvement of Residents' Open Surgical Skills

INTRODUCTION

The multiorgan procurement (MOP) represents a chance for the general surgery resident to learn the fundamental steps of open abdominal surgery. The objective of this study was to evaluate the impact of MOP on the residents' open surgical skills.

METHODS

Residents' surgical skills were assessed during a 6-month transplant rotation (October 2020-March 2021) using a modified Objective Structured Assessment of Technical Skills with the global rating scale. The surgeries were self-assessed by residents and tutors based on 9 specific steps (SS) and 4 general skills (GS). Each item was rated from 1 (poor) to 5 (excellent) with a maximum score of 45 points for SS and 20 for GS. A crossed-effects linear regression analysis was performed both to evaluate any associations between GS/SS scores and some prespecified covariates, and to study differences in the assessments performed by residents and tutors.

RESULTS

Residents actively participated in a total of 59 procurements. In general, there were no significant differences in SS/GS mean scorings between residents (n = 15) and tutors (n = 5). There was a significantly positive association between mean GS/SS scorings and the number of donor surgeries performed (at least 5). Comparing the evaluations of the tutors with the residents, this significance was retained only when scorings were assigned by the tutors.

CONCLUSIONS

MOP was shown to improve basic open surgical skills among residents. Awareness of the utility of a clinical rotation in transplant surgery should be raised also on an institutional level.

What are the learning objectives in surgical training - a systematic literature review of the surgical competence framework

OBJECTIVE

To map the landscape of contemporary surgical education through a competence framework by conducting a systematic literature review on learning outcomes of surgical education and the instructional methods applied to attain the outcomes.

BACKGROUND

Surgical education has seen a paradigm shift towards competence-based training. However, a gap remains in the literature regarding the specific components of competency taught and the instructional methods employed to achieve these outcomes. This paper aims to bridge this gap by conducting a systematic review on the learning outcomes of surgical education within a competence framework and the instructional methods applied. The primary outcome measure was to elucidate the components of competency emphasized by modern surgical curricula. The secondary outcome measure was to discern the instructional methods proven effective in achieving these competencies.

METHODS

A search was conducted across PubMed, Medline, ProQuest Eric, and Cochrane databases, adhering to PRISMA guidelines, limited to 2017-2021. Keywords included terms related to surgical education and training. Inclusion criteria mandated original empirical studies that described learning outcomes and methods, and targeted both medical students and surgical residents.

RESULTS

Out of 42 studies involving 2097 participants, most concentrated on technical skills within competency-based training, with a lesser emphasis on non-technical competencies. The effect on clinical outcomes was infrequently explored.

CONCLUSION

The shift towards competency in surgical training is evident. However, further studies on its ramifications on clinical outcomes are needed. The transition from technical to clinical competence and the creation of validated assessments are crucial for establishing a foundation for lifelong surgical learning.

How expert surgeons review robotic videos: A grounded theory study

BACKGROUND

Surgical video review has been shown to improve technical skill in novice surgeons; however, to date, there are no studies examining the use of video review by experts. This study sought to understand the process expert robotic surgeons use when they review their surgical videos.

METHODS

In this qualitative pilot study, eight expert robotic colorectal surgeons individually participated in semi-structured interviews and video elicitation sessions. Grounded theory was used to develop a process model on how expert surgeons review their robotic videos.

RESULTS

The participants identified four categories of video review: critical incidents, unique cases, new techniques/procedures, and routine cases. They ask themselves questions with the primary goal of improving surgical technical skills. Surgeons compare their performance to benchmark videos and share videos for teaching and collaboration.

CONCLUSIONS

Expert robotic surgeons use video review for self-reflection and self-assessment. They recognize that improving surgical skills requires a growth mindset. These findings may contribute to the surgical education of novices.

Implementation of an Online Intraoperative Assessment of Technical Performance for Surgical Trainees

INTRODUCTION

Assessment of surgical resident technical performance is an integral component of any surgical training program. Timely assessment delivered in a structured format is a critical step to enhance technical skills, but residents often report that the quality and quantity of timely feedback received is lacking. Moreover, the absence of written feedback with specificity can allow residents to seemingly progress in their operative milestones as a junior resident, but struggle as they progress into their postgraduate year 3 and above. We therefore designed and implemented a web-based intraoperative assessment tool and corresponding summary "dashboard" to facilitate real-time assessment and documentation of technical performance.

MATERIALS AND METHODS

A web form was designed leveraging a cloud computing platform and implementing a modified Ottawa Surgical Competency Operating Room Evaluation instrument; this included additional, procedure-specific criteria for select operations. A link to this was provided to residents via email and to all surgical faculty as a Quick Response code. Residents open and complete a portion of the form on a smartphone, then relinquish the device to an attending surgeon who then completes and submits the assessment. The data are then transferred to a secure web-based reporting interface; each resident (together with a faculty advisor) can then access and review all completed assessments.

RESULTS

The Assessment form was activated in June 2021 and formally introduced to all residents in July 2021, with residents required to complete at least one assessment per month. Residents with less predictable access to operative procedures (night float or Intensive Care Unit) were exempted from the requirement on those months. To date a total of 559 assessments have been completed for operations performed by 56 trainees, supervised by 122 surgical faculty and senior trainees. The mean number of procedures assessed per resident was 10.0 and the mean number per assessor was 4.6. Resident initiation of Intraoperative Assessments has increased since the tool was introduced and scores for technical and nontechnical performance reliably differentiate residents by seniority.

CONCLUSIONS

This novel system demonstrates that an online, resident-initiated technical assessment tool is feasible to implement and scale. This model's requirement that the attending enter performance ratings into the trainee's electronic device ensures that feedback is delivered directly to the trainee. Whether this aspect of our assessment ensures more direct and specific (and therefore potentially actionable) feedback is a focus for future study. Our use of commercial cloud computing services should permit cost-effective adoption of similar systems at other training programs.

Novices can assess microsurgery performance, and this is enhanced using the Manual Suture Parameters for Training and Assessment (M-SParTA) scoring system

This study explores how novices could effectively evaluate the quality of microsurgical suturing. That would be enhanced with using a novel Manual Suture Parameters for Training and Assessment (M-SParTA), which supported novices with guidance on the objective parameters, in order to increased the accuracy of scoring ability. We also propose the following initial framework to train novices in microsuturing using a standardised task: 1) Exposure; 2) Assessment; 3) Hands-on and self-assessment. The independent learning cycle with targeted supervision provides novices with greater autonomy and a less stressful environment that could enhance skills training.

Self-Assessment Versus Peer-Assessment in Microsurgery Learning: A Comparative Retrospective Study in a Surgery Residents Cohort

INTRODUCTION

In surgical learning, self-assessment allows the physician to identify and improve his strong and weak points. However, its scientific validity has yet to be demonstrated. The aim of this study was to analyze if there is a link between self-assessment accuracy and improvement in surgical skills. We make the hypothesis that an accurate self-assessment allows a greater improvement MATERIAL AND METHOD: We set up a retrospective cohort study at the tertiary University Hospital of Angers. Between 2019 and 2021, twenty-eight surgery residents took part into a microsurgery program and were included in the study. For two weeks, they performed anastomosis training on inert material and living anesthetized rats under microscope. Each resident was evaluated during the workshop by senior surgeons on 10 items: movement stability and fluidity, instrument manipulation, needles, dissection, clamp setting, vessel manipulation, suture, checking before clamp removal, checking after clamp removal, watertighness. Self-assessment was performed by the residents with the same grid, at the end of the workshop. Residents' and senior's evaluations were double-blind. We retrospectively analyzed the concordance between senior objective assessment and self-assessment, and the effect of an accurate self-assessment on technical improvement.

RESULTS

Data for twenty-five residents were analyzed, 14 were female (56%). The mean age was 29 years. Surgical specialties were orthopedics (44%), maxillofacial surgery (45.4%), neurosurgery (12%), gynecology (4%) and vascular surgery (4%). According to Cohen's kappa coefficient, 14 residents (56%) underestimated themselves, 7 (28%) were concordant with peer-assessment and 4 (16%) overestimated themselves. The concordance between self and peer assessment during sessions was positive for the most objective items, and negative for the most subjective items. Technical skills improvement in term of peer-assessment averages was positive for each item in each group, without statistical differences between groups.

CONCLUSION

We found that the ability to self-assess in a fast-track microsurgery module for surgery residents varied according to analyzed gestures. We demonstrated an improvement in term of self-assessment for objective items, and a decrease for subjective items. However, we didn't find any relation between improvement curve and the accuracy of self-assessment.

A Study of Surgical Residents' Self-Assessment of Open Surgery Skills Using Gap Analysis

BACKGROUND

Surgical residents need structured and objective feedback to develop their skills and become capable of performing surgical procedures autonomously. A shortage of experienced surgical staff has prompted residents to seek feedback from self-assessment and peer assessments.

OBJECTIVES

We investigated whether surgical residents can reliably rate their own and their peers' basic surgical skills using the Global Rating Scale (GRS) from the Objective Structured Assessment of Technical Skills.

METHODS

The study was a prospective and descriptive study conducted using gap analysis at the Copenhagen Academy for Medical Education and Simulation (Copenhagen, Denmark) from 2016 to 2017. Surgical residents were recruited during a course in basic open surgical skills. Among 102 course participants, 53 met the inclusion criteria and 22 participated in the study.

RESULTS

We recruited surgical residents based in the Capital Region and Zealand Region of Demark, and 42% of eligible residents participated in the study. Surgical residents underestimated their own surgical performance (median, 17 [range, 15-18] vs. 20 [range, 19.75-22]; P < 0.001). They also rated their peers lower than an experienced rater did (median 10 [range, 8.75-14] vs. 15 and median 20.5 [range, 19-22] vs. 23; both P < 0.001). Gap analysis revealed residents had unrecognized strengths (ie, self-underappraisal) in most GRS domains.

CONCLUSIONS

Ratings are unreliable when surgical residents assess their own and their peers' performances using GRS. A gap analysis revealed unrecognized strengths in time and motion, instrument handling, knowledge of instruments and sutures, and knowledge of specific procedure as well as unrecognized weaknesses in flow of operation and forward planning.

Quality Assessment and Modulating Factors on Self-Regulatory Behavior in Peer-Assisted Medical Learning

OBJECTIVES

Standardized extracurricular skills labs courses have been developed in recent decades and are important approaches in peer-assisted medical learning (PAL). To provide high quality training and achieve effective learning strategies, continuous evaluations and quality assessments are essential. This research aims to evaluate quality data from medical students participating in extracurricular skills labs courses at Ruhr-University Bochum to prospectively optimize concepts and didactical training and standardize processes. Additionally, we set out to assess and quantify drivers that are influencing factors of the self-reflection of competencies.

METHODS

The analysis was based on a routine assessment of n = 503 attendees of the PAL courses in the skills labs in three consecutive semesters, who voluntarily participated in the evaluation. We analyzed the effects of age, semester and their interaction on the self-reflection of competencies in technical skills courses using moderated regression and simple slope analyses, as previously published. A univariate analysis of variance (ANOVA) with post hoc Tukey HSD testing was used to analyze group means in estimated competencies using IBM SPSS Statistics V.28.

RESULTS

An analysis of variance revealed a significant increase in self-assessed competencies when comparing pre- vs. post-course evaluation data in all 35 depicted items (all p < 0.001). A total of 65.5% of the items were adjusted significantly differently, revealing modified self-reflected pre-course levels compared to those stated before. A moderated regression analysis revealed that age (R2 = 0.001, F(1;2347) = 1.88, p < 0.665), semester of study (∆R2 = 0.001, ∆F (1;2346) = 0.012, p < 0.912) and their interaction (∆R2 = 0.001, ∆F (1;2345) = 10.72, p < 0.227) did not explain a significant amount of the variance in self-reflection variance. A simple slope analysis of earlier (b = 0.07, t = 0.29, p < 0.773) and later semesters of study (b = 0.06, t = 0.07, p < 0.941) did not differentiate from zero.

CONCLUSIONS

The presented evaluation paradigm proved to be a useful tool to encourage students to initiate self-regulatory and self-reflective behavior. The cohesive evaluation of the large cohort of attendees in extracurricular, facultative skills labs courses was helpful in terms of quality assessments and future adaptations. Further evaluation paradigms should be implemented to assess other influencing factors, such as gender, on self-reflection, since age and semester did not explain significant differences in the model.

[The Importance of Feedback in Proctoring in Robotic Surgery - a Cornerstone of the Concept of Success?]

Robot-assisted surgical methods are a development and supplement to conventional laparoscopy. Thus, appropriate training of surgeons in this specialty is indispensable. Especially for the initial phase, simulation programs - similar to those used in aviation - are ideally suited to introduce surgeons to the technically challenging surgical procedure. Later on, but still at the beginning of the learning curve, proctoring has proven its worth in order to allow the surgeon to be trained on site, to train him individually and to introduce him to more complex cases. An essential component of successful training concepts is the feedback: after some time and in written form, the surgeon to be trained receives a summary or a guideline, which describes the existing situation as well as suggestions for improvement and future potential. The implementation of such feedback allows the surgeon, in addition to the number of cases completed, to put his self-assessment into perspective and adjust development goals. Thus, feedback is the crucial interface between the beginning of a learning curve and advanced surgical skills, including the possibility of realistic self-assessment.

Role of video self-assessment in laparoscopic simulation training: a randomized pilot trial

BACKGROUND

Residency programs have implemented simulation training to compensate for reduced operating room exposure. Video recording is an educational tool that can be utilized for coaching, telepresence, and self-assessment during simulation training. Data is limited on the utility of video recording and self-assessment for laparoscopic training in Ob/Gyn residency programs..

OBJECTIVE

This study aimed to determine the role of video self-assessment as an educational tool in laparoscopic simulation training and to establish the feasibility of our study design for a larger randomized controlled trial.

STUDY DESIGN

This was a prospective pilot study with a parallel, randomized, trial design that occurred in the Department of Obstetrics and Gynecology at the Mount Sinai Hospital. Subject participation took place in a surgical simulation training room. A total of 23 subjects were recruited (7 medical students, 15 residents, 1 fellow) voluntarily. All participants completed the study. All the subjects completed a pretest survey. The surgical simulation room contained a single Fundamentals of Laparoscopic Surgery box trainer and video-recording station. For session #1, each participant performed 2 Fundamentals of Laparoscopic Surgery tasks (A, peg transfer; B, intracorporeal knot tie). Participants were video recorded during session #1 and were randomized to either receive or not receive their video recording. The video group (n=13) and control group (n=10) repeated the Fundamentals of Laparoscopic Surgery tasks 7 to 10 days later (session #2). The primary outcome was percentage change in completion time between sessions. Secondary outcomes were percentage change in peg and needle drops between sessions.

RESULTS

The participant characteristics (video vs control) were as follows: average training level (6.15 vs 4.90 years), self-assessment (1=poor, 10=excellent) of surgical skill (4.8 vs 3.7), and laparoscopic skill (4.4 vs 3.5). Training level was inversely correlated with completion time for tasks A and B (r, -0.79 and -0.87; P<.0001). Less experienced trainees required the maximum time allotted for each task in session #1 (A, 3; B, 13). Regarding the primary outcome, the video group improved less than the control group (A, 16.7% vs 28.3%; B, 14.4% vs 17.3%). After controlling for training level (residents only), the video group improved more in the primary outcome (A, 17% vs 7.4%; B, 20.9% vs 16.5%) and secondary outcomes (A, 0.0% vs -194.1%; B, 41.3% vs 37.6%).

CONCLUSION

Video self-assessment has a potential role in simulation training for obstetrics-gynecology residents. With key improvements, the feasibility of our study design was demonstrated in preparation for a future definitive trial.

Video-based interventions to improve self-assessment accuracy among physicians: A systematic review

PURPOSE

Self-assessment of a physician's performance in both procedure and non-procedural activities can be used to identify their deficiencies to allow for appropriate corrective measures. Physicians are inaccurate in their self-assessments, which may compromise opportunities for self- development. To improve this accuracy, video-based interventions of physicians watching their own performance, an experts' performance or both, have been proposed to inform their self-assessment. We conducted a systematic review of the effectiveness of video-based interventions targeting improved self-assessment accuracy among physicians.

MATERIALS AND METHODS

The authors performed a systematic search of MEDLINE, Embase, EBM reviews, and Scopus databases from inception to August 23, 2022, using combinations of terms for "self-assessment", "video-recording", and "physician". Eligible studies were empirical investigations assessing the effect of video-based interventions on physicians' self-assessment accuracy with a comparison of self-assessment accuracy pre- and post- video intervention. We defined self-assessment accuracy as a "direct comparison between an external evaluator and self-assessment that was quantified using formal statistical analysis". Two reviewers independently screened records, extracted data, assessed risk of bias, and evaluated quality of evidence. A narrative synthesis was conducted, as variable outcomes precluded a meta-analysis.

RESULTS

A total of 2,376 papers were initially retrieved. Of these, 22 papers were selected for full-text review; a final 9 studies met inclusion criteria for data extraction. Across studies, 240 participants from 5 specialties were represented. Video-based interventions included self-video review (8/9), benchmark video review (3/9), and/or a combination of both types (1/9). Five out of nine studies reported that participants had inaccurate self-assessment at baseline. After the intervention, 5 of 9 studies found a statistically significant improvement in self-assessment accuracy.

CONCLUSIONS

Overall, current data suggests video-based interventions can improve self-assessment accuracy. Benchmark video review may enable physicians to improve self-assessment accuracy, especially for those with limited experience performing a particular clinical skill. In contrast, self-video review may be able to provide improvement in self-assessment accuracy for more experience physicians. Future research should use standardized methods of comparison for self-assessment accuracy, such as the Bland-Altman analysis, to facilitate meta-analytic summation.

Supplementing Intraoperative Mastoidectomy Teaching With Video-Based Coaching

OBJECTIVES

Video-based coaching might complement general surgery education, but little is known of its applicability for otologic microsurgical teaching. Our purpose was thus to evaluate the content and resident-perceived benefit of video-based coaching for mastoidectomy education.

METHODS

In this mixed-methods pilot design, mastoidectomies were recorded from operative microscopes and reviewed during 30-minute video-based coaching sessions at 2 tertiary care centers. Eight residents and 3 attendings participated. Ten-point Likert-type questionnaires on the extent to which attendings taught 12 topics through 8 techniques were completed by residents after surgical and coaching sessions. Coaching sessions and structured interviews with residents were audio-recorded, transcribed and iteratively coded.

RESULTS

Seven audio-recordings were available for coaching sessions, during which a mean of 2.22 ± 0.5 topics per minute were discussed. Of the 12 teaching topics, technique was discussed most frequently (32%, 0.71 ± 0.2 topics/min), followed by anatomy (16%, 0.31 ± 0.16 topics/min). Of all 8 ratings between coaching and operative sessions, residents indicated a greater extent of discussion of anatomy (median difference, [95% confidence interval (CI)] of 3 [1-4]), progress (2.25 [95% CI, 0.5-4]), technique (3.5 [95% CI, 1.5-5.5]), pitfalls (2.5 [95% CI, 1-3.5]), and summarizing (3 [95% CI, 1-5]). In structured interviews, residents reported improved self-confidence and global perspective.

CONCLUSIONS

Video-based coaching is educationally dense and characterized by perceived richer teaching and promotion of a deeper surgical understanding. It requires no additional resources, can be completed in a short period of time and can be implemented programmatically for any otolaryngologic subspecialty utilizing video-recording capable equipment.

Self-Assessment in Feedback Conversations: A Complicated Balance

PURPOSE

Learner-centered feedback models encourage educators to ask learners to self-assess at the start of feedback conversations. This study examines how learners perceive and respond to self-assessment prompts during feedback conversations and assesses medical students' perceptions of and approach to self-assessment used as the basis for these conversations.

METHOD

All rising second-, third-, and fourth-year medical students at a midwestern U.S. medical school were invited to participate in this study. Students participated in 1-on-1 interviews between June and August 2019 during which they were asked open-ended questions about their experiences with self-assessment and feedback during medical school. The interviews were audio recorded and transcribed, and comments related to self-assessment in feedback conversations were extracted. Thematic analysis was used to identify recurrent ideas and patterns within the transcripts, and all excerpts were reviewed and coded to ensure that the identified themes adequately captured the range of student responses.

RESULTS

A total of 25 students participated in the study. Although some students noted improvement in their self-assessment abilities with increasing experience, no consistent gender, race, or training-level differences were found in reported attitudes or preferences. Students identified many benefits of self-assessment and generally appreciated being asked to self-assess before receiving feedback. Students had varied responses to specific self-assessment prompts, with no clear preferences for any particular self-assessment questions. Students described weighing multiple factors, such as image concerns and worries about impact on subsequent evaluations, when deciding how to respond to self-assessment prompts.

CONCLUSIONS

The process by which learners formulate and share self-assessments in feedback conversations is not straightforward. Although educators should continue to elicit self-assessments in feedback discussions, they should recognize the limitations of these self-assessments and strive to create a safe environment in which learners feel empowered to share their true impressions.

Accuracy of self-assessment in gastrointestinal endoscopy: a systematic review and meta-analysis

BACKGROUND

Assessment is necessary to ensure both attainment and maintenance of competency in gastrointestinal (GI) endoscopy, and this can be accomplished through self-assessment. We conducted a systematic review with meta-analysis to evaluate the accuracy of self-assessment among GI endoscopists.

METHODS

This was an individual participant data meta-analysis of studies that investigated self-assessment of endoscopic competency. We performed a systematic search of the following databases: Ovid MEDLINE, Ovid EMBASE, Wiley Cochrane CENTRAL, and ProQuest Education Resources Information Center. We included studies if they were primary investigations of self-assessment accuracy in GI endoscopy that used statistical analyses to determine accuracy. We conducted a meta-analysis of studies using a limits of agreement (LoA) approach to meta-analysis of Bland-Altman studies.

RESULTS

After removing duplicate entries, we screened 7138 records. After full-text review, we included 16 studies for qualitative analysis and three for meta-analysis. In the meta-analysis, we found that the LoA were wide (-41.0 % to 34.0 %) and beyond the clinically acceptable difference. Subgroup analyses found that both novice and intermediate endoscopists had wide LoA (-45.0 % to 35.1 % and -54.7 % to 46.5 %, respectively) and expert endoscopists had narrow LoA (-14.2 % to 21.4 %).

CONCLUSIONS

GI endoscopists are inaccurate in self-assessment of their endoscopic competency. Subgroup analyses demonstrated that novice and intermediate endoscopists were inaccurate, while expert endoscopists have accurate self-assessment. While we advise against the sole use of self-assessment among novice and intermediate endoscopists, expert endoscopists may wish to integrate it into their practice.

Dunning-Kruger Effect Between Self-Peer Ratings of Surgical Performance During a MASCAL Event and Pre-Event Assessed Trauma Procedural Capabilities

Objectives

The research question asked to what extent do self-rated performance scores of individual surgeons correspond to assessed procedural performance abilities and to peer ratings of procedural performance during a mass casualty (MASCAL) event?

Background

Self-assessment using performance rating scales is ubiquitous in surgical education as a proxy for direct measurement of competence. The validity and reliability of self-ratings as competency measures are susceptible to cognitive biases such as Dunning-Kruger effects, which describe how individuals over/underestimate their own performance compared to assessments from independent sources. The ability of surgeons to accurately self-assess their procedural performance remains undetermined.

Methods

A purposive sample of military surgeons (N = 13) who collectively cared for trauma patients during a MASCAL event participated in the study. Pre-event performance assessment scores for 32 trauma procedures were compared with post-event self and peer performance ratings using F tests (P < 0.05) and effect sizes (Cohen's d).

Results

There were no significant differences between peer ratings and performance assessment scores. There were significant differences between self-ratings and both peer ratings (P < 0.001) and performance assessment scores (P < 0.001). Effect sizes were very large for self to peer rating comparison (Cohen's d = 2.34) and self to performance assessment comparison (Cohen's d = 2.77).

Conclusions

The outcomes demonstrate that self-ratings were significantly lower than the independently determined assessment scores for each surgeon, revealing a Dunning-Kruger effect for highly skilled individuals underestimating their abilities. These outcomes underscore the limitations of self-assessment for measuring competence.

Modelling the Effect of Age, Semester of Study and Its Interaction on Self-Reflection of Competencies in Medical Students

Objectives: Accurate self-assessment and -reflection of competencies are crucial skills for all health professions. The National Competence-Based Learning Objectives Catalogue (NKLM) guiding medical faculties in Germany points out reflection as a non-technical skill and competency-based medical education (CBME) as important approaches. In this context, the role and structure of curricula and skills labs evolved. Especially in peer-assisted trainings, reflection of competencies is important to improve self-regulated learning. Traditionally, we assume self-reflection skills to evolve automatically with learners’ experience. This approach aims to find empirical evidence for this assumption and implements self-reflection of competencies in clinical skills education. Here, we quantify the influence of age and semester of study and its interaction on the concordant self-reflection of students’ own competencies. Methods: Investigation was based on a retrospective analysis of evaluation data from peer-assisted “first aid” and “physical examination” courses in the skills labs of the medical faculty at the Ruhr-University Bochum, Germany. Participants were asked for self-assessed competencies before (pre) and after (post) the course. Additionally, they were asked to retrospectively re-rate their “before” competencies after completing the course (post-pre). Differences between pre and post-pre competencies were assessed as the concordant self-reflection in a moderated regression analysis. Group means and standard deviation were depicted using univariate analysis of variance (ANOVA) with post-hoc Tukey HSD testing in IBM SPSS Statistics V.28. Moderated regression and simple slope analyses were conducted to calculate interaction effects of age and semester of study on the concordant self-reflection. Results: As expected, participants (n = 168) showed significant progress in subjective self-assessment (pre vs. post) in all 18 assessed domains in the course (all p < 0.001). Additionally, participants self-assessed their previous competencies after the course (post-pre) differently than before the course (pre) in 11 out of 18 domains. Hereby, the interaction of age and semester of study explained a significant part of variance in the first aid course (∆R2 = 0.008, ∆F (1;1020) = 8.53, p < 0.005) and in the physical examination course (ΔR2 = 0.03, ΔF (1;10,280) = 10.72, p < 0.001). Conclusions: We quantified that interaction of age and semester has a significant influence on concordant self-reflection skills using a moderated regression analysis. Assumed as an indicator, we conclude that advanced and older students show less differences in pre- vs. post-pre-ratings. This has implications for curriculum development, postulating that an exposure to self-reflection as a metacognitive process should be introduced early in order to train competencies in health professionals. Prospective studies with competency-based assessments are necessary to validate findings.

Simulation-based Education Improves Military Trainees' Skill Performance and Self- Confidence in Tourniquet Placement: A Randomized Controlled Trial

BACKGROUND

Tactical Combat Casualty Care (TCCC) is the standard of care for stabilization and treatment of military trauma patients. The DoD has mandated that all Service members receive role-based TCCC training and certification. Simulation education can increase procedural skills by providing opportunities for deliberate practice in safe, controlled environments. We developed and evaluated the effectiveness of a simulation-based TCCC training intervention to improve participants' skill performance and self-confidence in tourniquet placement.

METHODS

This study was a single-blinded, randomized trial with waitlist controls. Army ROTC cadets from a single training battalion comprised the study population. After randomization and baseline assessment of all participants, Group A alone received focused, simulation-based TCCC tourniquet application training. Three months later, all participants underwent repeat testing, and after crossover, the waitlist Group B received the same intervention. Two months later, all cadets underwent a third/final assessment. The primary outcome was tourniquet placement proficiency assessed by total score achieved on a standardized 8-item skill checklist. A secondary outcome was self-confidence in tourniquet application skill as judged by participants' Likert scale ratings.

RESULTS

Forty-three Army ROTC cadets completed the study protocol. Participants in both Group A (n = 25) and Group B (n = 18) demonstrated significantly higher performance from baseline to final assessment at five months and two months, respectively, following the intervention. Mean total checklist score of the entire study cohort increased significantly from 5.53 (SD = 2.00) at baseline to 7.56 (SD = 1.08) at Time 3, a gain of 36.7% (p < 0.001). Both groups rated their self-confidence in tourniquet placement significantly higher following the training.

CONCLUSIONS

A simulation-based TCCC curriculum resulted in significant, consistent, and sustained improvement in participants' skill proficiency and self-confidence in tourniquet placement. Participants maintained these gains two to five months after initial training.

LEVEL OF EVIDENCE

Level II - RCT with significant difference and only one negative criterion (<80% follow-up).

Prehospital Airway Management Training and Education: An NAEMSP Position Statement and Resource Document

AbstractAirway management competency extends beyond technical skills to encompass a comprehensive approach to optimize patient outcomes. Initial and continuing education for airway management must therefore extend beyond a narrow focus on psychomotor skills and task completion to include appreciation of underlying pathophysiology, clinical judgment, and higher-order decision making. NAEMSP recommends:Active engagement in deliberate practice should be the guiding approach for developing and maintaining competence in airway management.EMS learners and clinicians must be educated in an escalating approach to airway management, where basic airway maneuvers form the central focus.Educational activities should extend beyond fundamental knowledge to focus on the development of clinical judgment.Optimization of patient outcomes should be valued over performance of individual airway management skills.Credentialing and continuing education activities in airway management are essential to advance clinicians beyond entry-level competency.Initial and continuing education programs should be responsive to advances in the evidence base and maintain adaptability to re-assess content and expected outcomes on a continual basis.

Comparing control room operators' and experts' assessment of team performance using structured task-specific observation protocols and scenario replay

Operators' self-assessment has received limited interest within process control or human-system evaluation. Research on self-assessment has been criticised for poor assessment methodology, and consequently, its status is unclear. This study hypothesised that, given adequate assessment methods (such as task-specific assessment items and scenario replay), we could observe relatively accurate self-assessment results. Eighteen licensed operators and two experts assessed team performance in six nuclear control room scenarios. The results reveal an overall agreement between operators and experts, measured by the intraclass correlation coefficient, ranging from 0.60 to 0.70, which lies close to the intraclass correlation coefficient of 0.75 for the experts. This demonstrates potential for achievement of relatively accurate operator self-assessment for complex work. The agreement varied in a similar manner for both expert agreement and operator-expert agreement across eight performance dimensions. In addition, the operators' self-assessment provided additional information beyond observer assessment in identifying non-acceptable performance items.

Feasibility and acceptability of virtually coaching residents on communication skills: a pilot study

BACKGROUND

Developing communication skills is a key competency for residents. Coaching, broadly accepted as a training modality in medical education, has been proven a successful tool for teaching communication skills. Little research is available thus far to investigate virtual coaching on communication skills for telemedicine encounters. The purpose of the study was to test the hypothesis that virtually coaching residents on communication skills is feasible and acceptable. We surveyed 21 resident-faculty pairs participating in a "fully virtual" coaching session (patient, coach, and resident were virtual).

METHODS

We asked 50 neurology resident-faculty coach pairs to complete one "fully virtual" coaching session between May 20 and August 31, 2020. After each session, the resident and coach completed a 15-item survey, including Likert-style scale and open-ended questions, assessing feasibility and acceptability. Descriptive statistics and qualitative content and thematic analyses were performed.

RESULTS

Forty-two percent (21/50) of all eligible residents completed "fully virtual" coaching sessions. The overall survey response rate was 91 % (38/42). The majority of respondents agreed that the direct observation and debriefing conversation were easy to schedule and occurred without technical difficulties and that debriefing elements (self-reflection, feedback, takeaways) were useful for residents. Ninety-five percent of respondents rated the coach's virtual presence to be not at all disruptive to the resident-patient interaction. Virtual coaching alleviated resident stress associated with observation and was perceived as an opportunity for immediate feedback and a unique approach for resident education that will persist into the future.

CONCLUSIONS

In this pilot study, residents and faculty coaches found virtual coaching on communication skills feasible and acceptable for telemedicine encounters. Many elements of our intervention may be adoptable by other residency programs. For example, residents may share their communication goals with clinic faculty supervisors and then invite them to directly observe virtual encounters what could facilitate targeted feedback related to the resident's goals. Moreover, virtual coaching on communication skills in both the in-person and telemedicine settings may particularly benefit residents in challenging encounters such as those with cognitively impaired patients or with surrogate decision-makers.

Evaluation of a Novel Laparoscopic Cholecystectomy Curriculum With the Use of Animal Models and Live Operating

BACKGROUND

Despite wide recognition of the benefits of simulation training, there is no established model for the teaching of laparoscopic cholecystectomy. The authors developed a replicable, intensive 2-day proficiency-based curriculum, to include simulation and live operating, designed to be practical to both attend and organize. The primary objective of this study was to evaluate this curriculum for improved procedural confidence, measured by participant self-evaluation. Secondary outcomes were objective improvement in technical skills in accordance with the Objective Structured Assessment of Technical Skills (OSATS) scale and trainee self-assessment scores.

METHODS

The course consisted of lectures, operating on a sheep hepatobiliary model through a laparoscopic box trainer, and live operating on female patient volunteers. It was attended by eight junior registrars. Precourse data collated included demographic information, experience, and procedural confidence scores using a visual analog scale. Performance on an animal model and live patient was assessed by experts using the OSATS score. Procedural confidence was re-evaluated after each task, as well as self-assessment of speed, accuracy, and overall performance.

RESULTS

Procedural confidence scores improved by a mean of 12% (P < 0.001). All trainees demonstrated sustained objective improvement in technical skills (P < 0.001). The overall mean OSATS score increased by 18%. Significant improvement was observed after performing the procedure on an animal model (P < 0.001); however, no further significant improvement was observed with live operating. No significant difference was found on trainee self-assessment scores, in any category.

CONCLUSIONS

The study describes a successful curriculum model for the teaching of laparoscopic cholecystectomy, to include procedural and technical skill acquisition, in addition to the refinement and development of procedural confidence. Importantly, this was carried out in a safe environment with direct transferability to the operating theater.

Are trainees working in obstetrics and gynecology confident and competent in the care of frail gynecological oncology patients?

INTRODUCTION

Older patients undergoing cancer surgery are at increased risk of post-operative complications, prolonged hospital stay, and mortality. Identification of frailty can help predict patients at high risk of peri-operative complications and allow a collaborative, multidisciplinary team approach to their care. A survey was conducted to assess the confidence and knowledge of trainees in obstetrics and gynecology regarding identification and management of peri-operative issues encountered in frail gynecological oncology patients.

METHODS

A web-based survey was distributed via the Audit and Research in Gynaecological Oncology (ARGO) collaborative and UK Audit and Research Collaborative in Obstetrics and Gynaecology (UKARCOG) . The survey on the management of frail peri-operative patients was disseminated to doctors-in-training (trainees) working in obstetrics and gynecology in the United Kingdom (UK) and Ireland. Specialty (ST1-7), subspecialty, and general practice trainees, non-training grade doctors, and foundation year doctors currently working in obstetrics and gynecology were eligible. Consultants were excluded. Study data were collected using REDCAP software hosted at the University of Manchester. Responses were collected over a 6-week period between January and February 2020.

RESULTS

Of the 666 trainees who participated, 67% (425/666) reported inadequate training in peri-operative management of frail patients. Validated frailty assessment tools were used by only 9% (59/638) of trainees and less than 1% (4/613) were able to correctly identify all the diagnostic features of frailty. Common misconceptions included the use of chronological age and gender in frailty assessments. The majority of trainees (76.5%, 448/586) correctly answered a series of questions relating to mental capacity; however, only 6% (36/606) were able to correctly identify all three diagnostic features of delirium. A total of 87% (495/571) of trainees supported closer collaboration with geriatricians and a multidisciplinary approach.

CONCLUSIONS

Obstetrics and gynecology trainees reported inadequate training in the peri-operative care of frail gynecological oncology patients, and overwhelmingly favored input from geriatricians. Routine use of validated frailty assessment tools may aid diagnosis of frailty in the peri-operative setting. There is an unmet need for formal education in the management of frail surgical patients within the UK and Irish obstetrics and gynecology curriculum.

"GRASP" Module of Self-assessment with Virtual Mentoring for Uninterrupted Surgical Training During COVID-19 Pandemic

Disruption in surgical training during the COVID-19 pandemic has prompted many innovative "virtual" modules to address the loss of learning exposure during these difficult times. We suggest "GRASP" (Gain, Recognize, Analyze, Simulate, and Perform) module of self-assessment with virtual mentoring for uninterrupted surgical training. This idea merges the advantages of self-assessment and mentoring for the benefit of surgical trainees. Its embedded continuous close individualized mentoring can change the surgical training culture by initiating an assessment of surgical learning and skills right from the beginning of surgical training.