Self-evaluation: how well do surgery residents judge performance on a rotation?

A national survey on the impact of the coronavirus pandemic on gynecologic surgical training

OBJECTIVE

The objective of this study was to assess the impact of the coronavirus pandemic on gynecology surgical training.

METHODS

A national cross-sectional online survey was distributed to all trainees and trainers in the higher specialist training program for obstetrics and gynecology in Ireland. The survey consisted of questions on topics which included: the volume of surgical procedures performed before and since the pandemic, confidence in performing various gynecologic procedures before and since the pandemic and questions regarding the impact of the pandemic on wellbeing and work practices.

RESULTS

Trainers and trainees experienced a significant reduction in operative volumes for most procedure types. Analysis showed a significant reduction in the number of minor procedures performed by trainees (z = -2.7, P = 0.007) and a significant reduction in the number of all procedure types performed by trainers (minor procedures z = -3.78, P = <0.001; intermediate procedures z = -4.48, P = < 0.001; major procedures z = -3.69, P = < 0.001). Respondents reported they had less time for research and audit, were less able to attend courses or conferences and worried about the impact of their work on their families.

CONCLUSIONS

In conclusion, this study has highlighted the current difficulties facing surgical trainees in gynecology because of the COVID-19 pandemic. These challenges have compounded an already challenging training environment for gynecology trainees. Efforts must be made to continue to provide high-quality tailored training to ensure the development of the next generation of gynecologic surgeons.

The Importance of Multiorgan Procurement in the Improvement of Residents' Open Surgical Skills

INTRODUCTION

The multiorgan procurement (MOP) represents a chance for the general surgery resident to learn the fundamental steps of open abdominal surgery. The objective of this study was to evaluate the impact of MOP on the residents' open surgical skills.

METHODS

Residents' surgical skills were assessed during a 6-month transplant rotation (October 2020-March 2021) using a modified Objective Structured Assessment of Technical Skills with the global rating scale. The surgeries were self-assessed by residents and tutors based on 9 specific steps (SS) and 4 general skills (GS). Each item was rated from 1 (poor) to 5 (excellent) with a maximum score of 45 points for SS and 20 for GS. A crossed-effects linear regression analysis was performed both to evaluate any associations between GS/SS scores and some prespecified covariates, and to study differences in the assessments performed by residents and tutors.

RESULTS

Residents actively participated in a total of 59 procurements. In general, there were no significant differences in SS/GS mean scorings between residents (n = 15) and tutors (n = 5). There was a significantly positive association between mean GS/SS scorings and the number of donor surgeries performed (at least 5). Comparing the evaluations of the tutors with the residents, this significance was retained only when scorings were assigned by the tutors.

CONCLUSIONS

MOP was shown to improve basic open surgical skills among residents. Awareness of the utility of a clinical rotation in transplant surgery should be raised also on an institutional level.

A national survey of surgical training in gynaecology: 2014-2021

OBJECTIVES

Over the last decade barriers to surgical training have been identified, including reducing access to theatre lists, reducing numbers of major surgical procedures being performed, increasing numbers of trainees and reduction in working hours since the introduction of the European Work Time Directive (EWTD). We aimed to assess the impact of these challenges on training in gynaecology over time.

STUDY DESIGN

We designed a study which aimed to assess both trainers and trainees perception of gynaecological surgical training in Ireland. The purpose of this was to identify confidence levels and challenges and to highlight potential areas for future improvement of surgical training in gynaecology. A a cross-sectional survey was distributed to all trainees and trainers registered with the Royal College of Physicians of Ireland Obstetrics and Gynaecology higher specialist training programme in 2014, 2017 and again in 2021.

RESULTS

During the study period trainees' confidence that the training programme prepared them to perform gynaecological surgery fell significantly. This fall in confidence was most evident for trainees' ability to perform abdominal hysterectomy (40.9% vs 15.2%, χ2 = 4.61, p =.03) and vaginal hysterectomy (31.8% vs 12.1%, χ2 = 4.58, p =.03) when comparing 2014 with 2021. All trainees reporteded that gynaecology was not given adequate time in the training programme to prepare them to practice independently as consultants. Themes identified by participants to improve training included dedicated access to theatre time with a named trainer, increased simulation training and subspecialisation at later stages of training.

CONCLUSION

Our findings show an overall decrease in trainees' and trainers' confidence in the surgical training available in gynaecology over an eight-year period.. This is particularly true for major gynaecology procedures. Efforts must be made to ensure trainees have improved access to surgical training in gynaecology. Potential solutions include improving access to simulation and incorporation of subspecialist training into later stages of training.

Are otolaryngology residents ready for independent practice? A survey study

OBJECTIVE

We surveyed otolaryngology program directors (PDs) and recent otolaryngology residency graduates on the operative autonomy of graduating residents and their comfort with independent practice.

METHODS

An anonymous survey was sent to otolaryngology PDs and recent graduates of training programs (members of the Young Physicians Section [YPS] of the American Academy of Otolaryngology-Head and Neck Surgery Foundation). Questions were developed around the 14 key indicator procedures (KIPs) defined by the Accreditation Council for Graduate Medical Education.

RESULTS

Fifty PDs (43% of PDs) and 152 recent graduates (6% of YPS members) responded. Over 90% of participating PDs felt their graduating residents were either somewhat or extremely comfortable performing 12 out of 14 KIPs. Among the 12 procedures PDs felt their graduating residents were comfortable performing, 57% to 95% of recent graduates also felt either somewhat or extremely comfortable performing them by graduation. Similarly, at least 90% of responding PDs felt their residents achieved meaningful autonomy in the last 2 months of residency prior to graduation for 11 of 14 KIPs. For these same 11 procedures, 74% to 95% of recent graduates indicated they achieved meaningful autonomy. The procedures that PDs and recent graduates felt required the most surgical assistance were ossiculoplasty/stapedectomy, rhinoplasty, and mastoidectomy. All PDs agreed or strongly agreed that graduating residents are comfortable operating and taking call as general otolaryngologists, compared to 86% and 93% of recent graduates.

CONCLUSION

Most PDs and recent graduates agree that residents are well-prepared for general otolaryngology practice with the exception of select KIPs.Level of evidence: 4.

Postgraduate Assessment Approaches Across Surgical Specialties: A Systematic Review of the Published Evidence

PURPOSE

Ensuring competence for surgical trainees requires holistic assessment of the qualities and competencies necessary to practice safely and effectively. To determine the next steps toward achieving this aim, the authors conducted a systematic review to summarize and appraise the available evidence related to any assessment approach to postgraduate surgical training and to identify the dominant themes for assessment approaches across different specialties or countries.

METHOD

Medline and Embase were searched on January 10, 2019, without language or time restrictions. Any peer-reviewed study that described an assessment framework (in practice or novel) throughout postgraduate surgical training globally was included. An iterative review and thematic analysis were performed on full-text articles to determine assessment themes. Studies were then grouped by assessment themes. A tailored quality assessment of the studies included in the final analysis was conducted. Assessment themes and validity were compared across surgical specialties and countries.

RESULTS

From an initial 7,059 articles, 91 studies (evaluating 6,563 surgical trainees) were included in the final analysis. Ten defined assessment themes were extracted. Ten studies (11.0%) were deemed low risk of bias based on the quality assessment tool used and thus were determined to be high quality. Minor differences in assessment themes were observed between specialties and countries. Assessment themes neglected by individual surgical specialties and assessment themes that need validated assessment tools were identified.

CONCLUSIONS

This review highlights the low quality of evidence and fragmented efforts to develop and optimize surgical assessments. The minor differences observed demonstrate a common approach, globally and across specialties, related to surgical assessments. A paradigm shift in assessment approaches, which will require national and international collaboration, is required to optimize design and validation so that a comprehensive assessment of surgical competence can be implemented.

Evidence That Female Urologists and Urology Trainees Tend to Underrate Surgical Skills on Self-Assessment

BACKGROUND

Historically low, the proportion of female urology residents now exceeds 25% in recent years. Self-assessment is a widely used tool to track progress in medical education. However, the validity of its results and gender differences may influence interpretation. Simulation of surgical skills is increasingly common in modern residency training and standardizes certain objective tasks and skills. The objective of this study was to identify gender differences in self-assessment of surgeons and trainees when using simulation of surgical skills.

METHODS

Medical students, residents, and attending and retired surgeons completed simple interrupted suturing. Assessment was self-rated using previously tested visual analog motion scales. Tasks were video recorded and rated by blinded expert surgeons using identical motion scales. Computer vision motion tracking software was used to objectively analyze the kinematics of surgical tasks.

RESULTS

Proportion of female (n = 17) and male (n = 20) participants did not differ significantly by the level of training, P = 0.76. Five expert surgeons evaluated 84 video segments of simple interrupted suturing tasks (mean 3.0 segments per task per participant). Self-assessment correlated well overall with expert rating for motion economy (Pearson correlation coefficient 0.61, P < 0.001) and motion fluidity (0.55, P = 0.002). Women underrated their performance in accordance with mean individual difference of self-assessment and expert assessment scores (Δ SAS-EAS) for both economy of motion (mean ± SEM -1.1 ± 0.38, P = 0.01) and fluidity of motion (-1.3 ± 0.39, P < 0.01). On the same measures, men tended to rate themselves in accordance with experts (-0.16 ± 0.36, P = 0.63; -0.09 ± 0.41, P = 0.82, respectively). Δ SAS-EAS did not differ significantly on any rating scale across levels of training. Expert ratings did not differ significantly by gender for any domain.

CONCLUSIONS

Female surgeons and trainees underrate some technical skills on self-assessment when compared with expert ratings, whereas male surgeon and trainee self-ratings and expert ratings were similar. Further work is needed to determine if these differences are accentuated across increasingly difficult tasks.

Instruments to measure the ability to self-reflect: A systematic review of evidence from workplace and educational settings including health care

INTRODUCTION

Self-reflection has become recognised as a core skill in dental education, although the ability to self-reflect is valued and measured within several professions. This review appraises the evidence for instruments available to measure the self-reflective ability of adults studying or working within any setting, not just health care.

MATERIALS AND METHODS

A systematic review was conducted of 20 electronic databases (including Medline, ERIC, CINAHL and Business Source Complete) from 1975 to 2017, supplemented by citation searches. Data were extracted from each study and the studies graded against quality indicators by at least two independent reviewers, using a coding sheet. Reviewers completed a utility analysis of the assessment instruments described within included studies, appraising their reported reliability, validity, educational impact, acceptability and cost.

RESULTS

A total of 131 studies met the inclusion criteria. Eighteen were judged to provide higher quality evidence for the review and three broad types of instrument were identified, namely: rubrics (or scoring guides), self-reported scales and observed behaviour.

CONCLUSIONS

Three types of instrument were identified to assess the ability to self-reflect. It was not possible to recommend a single most effective instrument due to under reporting of the criteria necessary for a full utility analysis of each. The use of more than one instrument may therefore be appropriate dependent on the acceptability to the faculty, assessor, student and cost. Future research should report on the utility of assessment instruments and provide guidance on what constitutes thresholds of acceptable or unacceptable ability to self-reflect, and how this should be managed.

Building a Framework for Self-Regulated Learning in Surgical Education: A Delphi Consensus Among Experts in Surgical Education

OBJECTIVE

Practice-Based Learning and Improvement is a Core Competency for surgical residents. Self-regulated learning (SRL) skills are an important component of this competency, yet are rarely taught in surgical training. Before we can teach SRL skills to residents we must understand the attributes that are essential. The purpose of this study was to develop a framework for SRL for surgical trainees.

DESIGN

This mixed-methods study design utilized a two-round modified-Delphi approach to develop consensus among experts in surgical education regarding SRL in surgical training. Round One included SRL constructs derived from educational, professional, and medical literature. Based upon quantitative data and thematic coding of comments, these constructs were adapted for applicability in the context of surgical residency and reorganized using a constant comparative approach. Revised constructs and groupings were presented to the expert panel in Round Two. Further survey rounds were not needed as all items in Round Two reached the predetermined consensus level of 70%.

SETTING

The Delphi panel was a purposeful sample of nationally recognized experts in surgical education, including members of the Association for Surgical Education and the Association for Program Directors in Surgery.

PARTICIPANTS

Thirty-eight of 42 experts (90.5%) responded to Round One, representing 29 academic and community medical systems nationally. The response rate for Round Two was 92%, 35 of the 38 Round One participants.

RESULTS

In Round One, the SRL constructs were all viewed as important with median scores ranging from 50 to 99.5, on a 100-point scale. Two hundred and ninety-one comments were coded and used to refine SRL definitions into 7 domains for Round Two, which included self-awareness, task analysis, situation awareness, strategic planning, progress evaluation, learning and performance management, and goal attainment and refinement. All Round Two items reached greater than 70% agreement, and received 51 free response comments. Several key themes emerged: clinical prioritization over learning, learner's limited control, value and reliance on external resources, low use of metacognition, and complex goal orientation. Incorporation of common themes generated a novel multi-stage framework of SRL in surgical education.

CONCLUSIONS

Surgical residency represents a unique learning context, in which the ideal learner is one who understands their learning environment and utilizes available resources to optimize their own learning. Experts in surgical education believe SRL skills are important in training, and a novel framework of SRL is necessary to support a learner-centered model within the demanding environment of surgical training.

Targeted surgical coaching can improve operative self-assessment ability: A single-blinded nonrandomized trial

BACKGROUND

The aim of the study was to investigate the effect of targeted surgical coaching on self-assessment of laparoscopic operative skill. Accurate self-assessment is vital for autonomous professional development. Surgical coaching can be used for performance improvement, but its role in this domain has been insufficiently investigated.

METHODS

This was a single site, nonrandomized, interrupted time series design trial. Participants were residents, fellows, and attending surgeons regularly performing laparoscopic general surgery operations. Each participant was enrolled in an individualized coaching program using review of personal and peer laparoscopic videos. The program involved 3 to 5 sessions over a period of 6 to 19 weeks. Coaching used case debriefing to target self-assessment proficiency, with a focus on objective interpretation of observations and facilitative capacity building. The primary outcome measure was self-assessment accuracy and correlation to expert ratings. The Objective Structured Assessment of Technical Skill global rating scale was utilized for evaluation.

RESULTS

Twelve participants were recruited and completed the coaching program. At baseline, there was no correlation between self-assessment and expert ratings. After completion of the coaching program there was correlation between self-assessment and expert ratings (P = .003) and improved self-assessment accuracy compared to baseline (P = .041).

CONCLUSION

This study has demonstrated that targeted coaching using video review of laparoscopic cases can improve operative self-assessment accuracy using the Objective Structured Assessment of Technical Skill.

Ability of Ophthalmology Residents to Self-Assess Their Performance Through Established Milestones

OBJECTIVES

Accurate self-assessment is an important aspect of practice-based learning and improvement and a critical skill for resident growth. The Accreditation Council for Graduate Medical Education mandates semiannual milestones assessments by a clinical competency committee (CCC) for all ophthalmology residents. There are six core competencies: patient care (PC), medical knowledge, systems-based practice, practice-based learning and improvement, professionalism, and interpersonal communication skills. These competencies are assessed by the milestones rubric, which has detailed behavioral anchors and are also used for trainee self-assessments. This study compares resident self-assessed (SA) and faculty CCC milestones scores.

DESIGN

Residents completed milestones self-assessments prior to receiving individual score reports from the CCC. Correlation coefficients were calculated comparing the SA and CCC scores. In addition, statistical models were used to determine predictors of disparities and differences between the SA and CCC scores.

SETTING

Wilmer Eye Institute, Johns Hopkins Hospital.

PARTICIPANTS

Twenty-one residents in the Wilmer Ophthalmology Residency program from July 2014 to June 2016.

RESULTS

Fifty-seven self-assessments were available for the analysis. For each resident's first assessment, SA and CCC scores were strongly correlated (r ≥ 0.6 and p < 0.05) for four milestones, and not correlated for the remaining 20 milestones. In multivariable models, the SA and CCC scores are less disparate for medical knowledge and systems-based practice competencies compared to practice-based learning and improvement. Higher year of training, PC and professionalism competencies were predictive of statistically significant resident overestimation of scores relative to the CCC. In addition, higher CCC scores predicted statistically significant lower SA-CCC disparities and differences. SA-CCC differences did not lower to a significant extent with repeated assessments or modification to the end-of-rotation evaluation forms.

CONCLUSIONS

Self-assessments by ophthalmology residents are not well-correlated with faculty assessments, emphasizing the need for improved and frequent timely feedback. Residents have the greatest difficulty self-assessing their professionalism and PC competency. In general, senior residents and underperforming residents have more inaccurate self-assessments.

Ready for Residency: A Bloomian Analysis of Competency-Based Osteopathic Medical Education

Context

Bloom's Taxonomy is a widely accepted tool for analyzing learning objectives, creating assessment materials, and ensuring that students move progressively through various levels of knowledge and cognition. Competency-based osteopathic medical education has, to the authors' knowledge, yet to be subjected to systematic bloomian analysis.

Objective

To advance a bloomian analysis of competencies published by the American Association of Colleges of Osteopathic Medicine (AACOM) and draw consequences of that analysis for undergraduate osteopathic medical education.

Methods

Content analysis of the verbs and contexts used in the AACOM competencies was conducted, followed by categorization by the various cognitive and knowledge dimensions as specified by the Revised Bloom's Taxonomy.

Results

The majority of AACOM competencies gravitate toward the taxonomy's apply and procedural cognitive and knowledge dimensions, respectively. In addition, 58% of the top 10 most-used verbs in the document were identified to be in the apply cognitive dimension. An analysis of the competencies revealed that 69% can be categorized under the procedural knowledge dimension. Of 657 code co-occurances, 45% were from the apply cognitive and procedural knowledge dimensions. These figures suggest a skewing toward application-based cognitive and procedural-based knowledge over other types of cognitive and knowledge dimensions.

Conclusion

Bloom's Taxonomy offers a useful framework for understanding how competency-based osteopathic medical education is organized. Such analysis underscores the importance of attending to the language and construction of competencies to better understand how the language of competencies shapes pedagogic practices and begin a broader conversation about the appropriate use and distribution of lower- and higher-order cognitive processes within competency-based osteopathic medical education.

A Novel Approach to Assessing Professionalism in Preclinical Medical Students Using Multisource Feedback Through Paired Self- and Peer Evaluations

Phenomenon: Professionalism is integral to the role of the physician. Most professionalism assessments in medical training are delayed until clinical rotations where multisource feedback is available. This leaves a gap in student assessment portfolios and potentially delays professional development.

APPROACH

A total of 246 second-year medical students (2013-2015) completed self- and peer assessments of professional behaviors in 2 courses following a series of Team-Based Learning exercises. Correlation and regression analyses were used to examine the alignment or misalignment in the relationship between the 2 types of assessments. Four subgroups were formed based on observed patterns of initial self- and peer assessment alignment or misalignment, and subgroup membership stability over time was assessed. A missing data analysis examined differences between average peer assessment scores as a function of selective nonparticipation.

FINDINGS

Spearman correlation demonstrated moderate to strong correlation between self-assessments completed alone (no simultaneous peer assessment) and self-assessments completed at the time of peer assessments (ρ = .59, p < .0001) but weak correlation between the two self-assessments and peer assessments (alone: ρ = .13, p < .013; at time of peer: ρ = .21, p < .0001). Generalized estimating equation models revealed that self-assessments done alone (p < .0001) were a significant predictor of self-assessments done at the time of peer. Course was also a significant predictor (p = .01) of self-assessment scores done at the time of peer. Peer assessment score was not a significant predictor. Bhapkar's test revealed subgroup membership based on the relationship between self- and peer ratings was relatively stable across Time 1 and Time 2 assessments (χ² = 0.83, p = .84) for all but one subgroup; members of the subgroup with initially high self-assessment and low peer assessment were significantly more likely to move to a new classification at the second measurement. A missing data analysis revealed that students who completed all self-assessments had significantly higher average peer assessment ratings compared to students who completed one or no self-assessments with a difference of -0.32, 95% confidence interval [-0.48, -0.15]. Insights: Multiple measurements of simultaneous self- and peer assessment identified a subgroup of students who consistently rated themselves higher on professionalism attributes relative to the low ratings given by their peers. This subgroup of preclinical students, along with those who elected to not complete self-assessments, may be at risk for professionalism concerns. Use of this multisource feedback tool to measure perceptual stability of professionalism behaviors is a new approach that may assist with early identification of at-risk students during preclinical years.

Does One Size Fit All? Examining the Application of Neurosurgery Residency Milestones Developed in the United States to a Taiwanese Culture

BACKGROUND

The Milestone Project was launched in 2009, charging specialties to develop specific educational accomplishments required to establish clinical competency. The milestone assessment method was first introduced to Taiwan in 2013 and before applying milestone assessments to our medical education system, the validity and reliability of these questionnaires needed to be evaluated.

METHODS

Twenty neurosurgical faculty members representing 3 clinical divisions and all 4 branch institutes completed milestone questionnaires for 26 residents semiannually, resulting in 435 resident assessments being collected and analyzed.

RESULTS

Cronbach's α, Kuder-Richardson Formula 20, and Kendall's W were used to show acceptable reliability and validity. Rater consistencies for nonskilled parts found that rater consistency progressively improved with time. Not all raters were able to assess the residents for the skilled parts resulting in nonassessable rates ranging from 9.5%-89.4%. For nonskilled and skilled items, milestone level as assessed by the staff improved as the resident progressed from residency year 3 (R3) to R6 in the residency program and showed that the milestone achievement level for an R3 was lower than that of an R6.

CONCLUSIONS

Milestone assessments have high reliability and may be a helpful assessment tool. Although milestone assessment can provide thorough feedback concerning performance and the content of the training program, they may not perfectly suit all residency-training programs, especially in different countries or different cultures. Modifications should be done before applying milestones to different areas; therefore the results can truly reflect the progress and condition of the training and learning process.

A pilot study of orthopaedic resident self-assessment using a milestones' survey just prior to milestones implementation

OBJECTIVE

To pilot test if Orthopaedic Surgery residents could self-assess their performance using newly created milestones, as defined by the Accreditation Council on Graduate Medical Education.

METHODS

In June 2012, an email was sent to Program Directors and administrative coordinators of the 154 accredited Orthopaedic Surgery Programs, asking them to send their residents a link to an online survey. The survey was adapted from the Orthopaedic Surgery Milestone Project. Completed surveys were aggregated in an anonymous, confidential database. SAS 9.3 was used to perform the analyses.

RESULTS

Responses from 71 residents were analyzed. First and second year residents indicated through self-assessment that they had substantially achieved Level 1 and Level 2 milestones. Third year residents reported they had substantially achieved 30/41, and fourth year residents, all Level 3 milestones. Fifth year, graduating residents, reported they had substantially achieved 17 Level 4 milestones, and were extremely close on another 15. No milestone was rated at Level 5, the maximum possible. Earlier in training, Patient Care and Medical Knowledge milestones were rated lower than the milestones reflecting the other four competencies of Practice Based Learning and Improvement, Systems Based Practice, Professionalism, and Interpersonal Communication. The gap was closed by the fourth year.

CONCLUSIONS

Residents were able to successfully self-assess using the 41 Orthopaedic Surgery milestones. Respondents' rate improved proficiency over time. Graduating residents report they have substantially, or close to substantially, achieved all Level 4 milestones. Milestone self-assessment may be a useful tool as one component of a program's overall performance assessment strategy.

Verification of accurate technical insight: a prerequisite for self-directed surgical training

Simulation-based surgical skills training during preclinical education is a persistent challenge due to time constraints of trainees and instructors alike. Self-directed practice is resource-efficient and flexible; however, insight into technical proficiency among trainees is often lacking. The purpose of this study is to prospectively assess the accuracy of self-assessments among medical students learning basic surgical suturing. Over seven weekly practice sessions, preclinical medical students performed serial repetitions of a simulation-based suturing task under one-on-one observation by one of four trainers. Following each task repetition, self- and trainer-assessments (SA-TA) were performed using a 36-point weighted checklist of technical standards developed a priori by expert consensus. Upon study completion, agreement between SA and TA was measured using weighted Cohen's kappa coefficients. Twenty-nine medical students each performed a median of 25 suture task repetitions (IQR 21.5-28). Self-assessments tended to overestimate proficiency during the first tertile of practice attempts. Agreement between SA and TA improved with experience, such that the weighted kappa statistics for the two-handed and instrument ties were >0.81 after 18-21 task attempts. Inexperienced trainees frequently overestimate technical proficiency through self-assessments. However, this bias diminishes with repetitive practice. Only after trainees have attained the capacity to accurately self-assess can effective self-directed learning take place.