Challenges in measuring ACGME competencies: considerations for milestones

Cumulative Sum Chart as Complement to Objective Assessment of Graduating Surgical Resident Competency: An Exploratory Study

BACKGROUND

Rater-based assessment and objective assessment play an important role in evaluating residents' clinical competencies. We hypothesize that a cumulative sum (CUSUM) chart of operative time is a complement to the assessment of chief general surgery residents' competencies with ACGME Milestones, aiding residency programs' determination of graduating residents' practice readiness.

STUDY DESIGN

We extracted ACGME Milestone evaluations of performance of operations and procedures (POP) and 3 objective metrics (operative time, case type, and case complexity) from 3 procedures (cholecystectomy, colectomy, and inguinal hernia) performed by 3 cohorts of residents (N = 15) during their PGY4-5. CUSUM charts were computed for each resident on each procedure type. A learning plateau was defined as at least 4 cases consistently locating around the centerline (target performance) at the end of a CUSUM chart with minimal deviations (range 0 to 1).

RESULTS

All residents reached the ACGME graduation targets for the overall POP by the end of chief year. A total of 2,446 cases were included (cholecystectomy N = 1234, colectomy N = 507, and inguinal hernia N = 705), and 3 CUSUM chart patterns emerged: skewed distribution, bimodal distribution, and peaks and valleys distribution. Analysis of CUSUM charts revealed surgery residents' development processes in the operating room towards a learning plateau vary, and only 46.7% residents reach a learning plateau in all 3 procedures upon graduation.

CONCLUSIONS

CUSUM charts of operative time complement the ACGME Milestones evaluations. The use of both may enable residency programs to holistically determine graduating residents' practice readiness and provide recommendations for their upcoming career/practice transition.

Reimagining a pass/fail clinical core clerkship: a US residency program director survey and meta-analysis

Pass/fail (P/F) grading has emerged as an alternative to tiered clerkship grading. Systematically evaluating existing literature and surveying program directors (PD) perspectives on these consequential changes can guide educators in addressing inequalities in academia and students aiming to improve their residency applications. In our survey, a total of 1578 unique PD responses (63.1%) were obtained across 29 medical specialties. With the changes to United States Medical Licensure Examination (USMLE), responses showed increased importance of core clerkships with the implementation of Step 2CK cutoffs. PDs believed core clerkship performance was a reliable representation of an applicant's preparedness for residency, particularly in Accreditation Council for Graduate Medical Education's (ACGME)Medical Knowledge and Patient Care and Procedural Skills. PDs disagreed with P/F core clerkships because it more difficult to objectively compare applicants. No statistically significant differences in responses were found in PD preferential selection when comparing applicants from tiered and P/F core clerkship grading systems. If core clerkships adopted P/F scoring, PDs would further increase emphasis on narrative assessment, sub-internship evaluation, reference letters, academic awards, professional development and medical school prestige. In the meta-analysis, of 6 studies from 2,118 participants, adjusted scaled scores with mean difference from an equal variance model from PDs showed residents from tiered clerkship grading systems overall performance, learning ability, work habits, personal evaluations, residency selection and educational evaluation were not statistically significantly different than from residents from P/F systems. Overall, our dual study suggests that while PDs do not favor P/F core clerkships, PDs do not have a selection preference and do not report a difference in performance between applicants from P/F vs. tiered grading core clerkship systems, thus providing fertile grounds for institutions to examine the feasibility of adopting P/F grading for core clerkships.

Shedding Light on the Hidden Curriculum: A Systematic Review of Ethics Education during General Surgery and Subspecialty Training

INTRODUCTION

Ethics education in surgical residencies is often delivered in an informal and nonstructured manner as part of a "hidden curriculum", leading to few residencies routinely including it in their core curriculum. This systematic review aimed to summarize the delivery modes, curriculum, structure, and effectiveness of ethics educational interventions for surgical trainees.

METHODS

We performed a comprehensive database search including MEDLINE, Embase, Scopus and CENTRAL to search for articles describing the implementation of ethics educational interventions for general surgery or subspecialty trainees.

RESULTS

A total of 14 studies were included. Only 2 centers performed targeted needs assessment. Curricula covered included informed consent, the doctor-patient relationship, breaking bad news, decision-making, end-of-life care, conflicts of interest, considering patients' personal contexts, and surgical research ethics. Modes of delivery varied across studies, including case-based learning, group discussions, didactic lectures, reading material, role-playing, simulated patients, and ethics morbidity and mortality (M&M) meetings. Evaluations were most commonly via surveys exploring knowledge and self-reported confidence, with only 3 studies measuring actual trainee performance using objective structured clinical examinations. In general, the educational interventions increased trainees' confidence/preparedness in handling ethical dilemmas.

CONCLUSION

We recommend comprehensive local needs assessment to guide curricular development and designing specific learning objectives and measurable milestones to ensure evaluation. Educational interventions are best delivered in a graduated manner with the complexity of the topic mirroring residents' real-life clinical responsibilities and experiences. Teaching modalities should be tailored according to the nature of the curricular content being taught to make the learning experience more interactive and effective.

Nontechnical Skills for Surgeons as a Framework to Evaluate Cardiopulmonary Bypass Management Skills of Resident Trainees

BACKGROUND

Nontechnical skills are critical in cardiac surgery but currently there is no formal paradigm to teach these in residency training. We investigated the use of the Nontechnical skills for surgeons (NOTSS) system as a framework to assess and teach nontechnical skills related to cardiopulmonary bypass (CPB) management.

METHODS

Single-center retrospective analysis of Integrated and Independent pathway thoracic surgery residents who participated in dedicated nontechnical skills evaluation and training. Two CPB management simulation scenarios were utilized. All residents received a lecture on CPB fundamentals and then individually participated in the first simulation ("Pre-NOTSS"). Immediately following this, nontechnical skills were rated by self-assessment and by a NOTSS trainer. All residents then underwent group NOTSS training followed by the second individual simulation ("Post-NOTSS"). Nontechnical skills were rated as before. NOTSS categories assessed included Situation Awareness, Decision Making, Communication and Teamwork, and Leadership.

RESULTS

Nine residents were divided into 2 groups: Junior (n = 4, PGY1-4) and Senior (n = 5, PGY5-8). Pre-NOTSS resident self-ratings were higher for Senior than Junior in the categories of Decision Making, Communication and Teamwork, and Leadership while trainer ratings were similar between the groups. Post-NOTSS, resident self-ratings were higher for Senior than Junior in Situation Awareness and Decision Making while trainer scores were higher for both groups in Communication and Teamwork and Leadership.

CONCLUSIONS

The NOTSS framework in conjunction with simulation scenarios provides a practical framework to evaluate and teach nontechnical skills related to CPB management. NOTSS training can lead to improvements in both subjective and objective ratings of nontechnical skills for all PGY levels.

Assessment of neurosurgical resident milestone evaluation reporting and feedback processes

OBJECTIVE

Structured performance evaluations are important for the professional development and personal growth of resident learners. This process is formalized by the Accreditation Council for Graduate Medical Education milestones assessment system. The primary aim of this study was to understand the current feedback delivery mechanism by exploring the culture of feedback, the mechanics of delivery, and the evaluation of the feedback itself.

METHODS

Face-to-face interviews were conducted with 10 neurosurgery residents exploring their perceptions of summative feedback. Coded data were analyzed qualitatively for overriding themes using the matrix framework method. A priori themes of definition of feedback, feedback delivery, and impact of feedback were combined with de novo themes discovered during analysis.

RESULTS

Trainees prioritized formative over summative feedback. Summative and milestone feedback were criticized as being vague, misaligned with practice, and often perceived as erroneous. Barriers to implementation of summative feedback included perceived veracity of feedback, high interrater variability, and the inconstant adoption of a developmental progression model. Gender bias was noted in degree of feedback provided and language used.

CONCLUSIONS

Trainee perception of feedback provided multiple areas of improvement. This paper can serve as a baseline to study improvements in the milestone feedback process and optimize learning.

Association Between Clinical Competencies and Mental Health Symptoms Among Frontline Medical Staff During the COVID-19 Outbreak: A Cross-Sectional Study

Background

In China, mental health of frontline medical staff might be influenced by clinicians' ability to handle the outbreak of coronavirus disease 2019 (COVID-19). Few studies to-date have addressed the association between clinicians' competencies and mental health in this context. This cross-sectional study was to examine the prevalence of mental health symptoms among frontline medical staff that fought against the COVID-19 outbreak, and explore the associations between their competencies, and separate and concurrent depressive and anxiety symptoms.

Methods

A total of 623 frontline medical staff was included in this study. Competencies, depressive symptoms, and anxiety symptoms were assessed using a self-reported short form of the Chinese clinical physicians' competency model, Patient Health Questionnaire-9, and Generalized Anxiety Disorder-7 questionnaire, respectively. Logistic regression models were used to evaluate the associations between one SD increase in competency scores and the prevalence of mental health problems.

Results

The prevalence of depressive, anxiety, and comorbid depressive and anxiety symptoms was 40.93, 31.78, and 26.00%, respectively. Among the medical staff with higher total competency scores, the prevalence of depressive [odds ratios (ORs) = 0.67, 95% confidence intervals (CIs): 0.55-0.81], anxiety (OR = 0.68, 95% CI: 0.56-0.83), and comorbid anxiety and depressive symptoms (OR = 0.69, 95% CI: 0.55-0.83) was lower than among their lower-scoring counterparts. Subgroup analyses stratified by core competency scores revealed similar associations as the main analyses.

Conclusion

The present findings highlight the association between high core competency scores and lower prevalence of depressive, anxiety, and comorbid anxiety and depressive symptoms.

Implementation of a pilot novel objective peer comparison evaluation system in an emergency medicine residency program

OBJECTIVES

Emergency medicine (EM) residents are currently evaluated via The Milestones, which have been shown to be imperfect and subjective. There is also a need for residents to achieve competency in patient safety and quality improvement processes, which can be accomplished through provision of peer comparison metrics. This pilot study aimed to evaluate the implementation of an objective peer comparison system for metrics that quantified aspects of quality and safety, efficiency and throughput, and utilization.

METHODS

This pilot study took place at an academic, tertiary care center with a 3-year residency and 14 residents per postgraduate year (PGY) class. Metrics were compared within each PGY class using Wilcoxon signed-rank and rank-order analyses.

RESULTS

Significant changes were seen in the majority of the metrics for all PGY classes. PGY3s accounted for the significant change in EKG and X-ray reads, while PGY1s and PGY2s accounted for the significant change in disposition to final note share. Physician evaluation to disposition decision was the only metric that did not reach significance in any class.

CONCLUSIONS

These preliminary data suggest that providing objective metrics is possible. Peer comparison metrics could provide an effective objective addition to the milestone evaluation system currently in use.

The telemedicine experience: using principles of clinical excellence to identify disparities and optimize care

The use of telemedicine has increased significantly during the Corona virus disease 2019 pandemic. This manuscript serves to identify the underlying principles of clinical excellence in telemedicine and to determine whether effective care practices can be generalized as a one-size-fits-all model or must instead be tailored to individual patient populations.A survey assessing care quality and patient satisfaction for patients using telemedicine was created and administered via email to 2 urban cohorts of varying demographics and socioeconomic backgrounds: a population of patients with prior stroke and cerebrovascular disease, and a cohort of patients followed for interstitial lung disease. Results were compared across groups to determine the generalizability of effective practices across populations.Individuals taking part in telemedicine were more likely to be White, more affluent, and woman, regardless of clinical diagnosis compared with a similar cohort of patients seen in-person the year prior. A lower-than-expected number of patients who were Black and of lower socioeconomic status followed up virtually, indicating potential barriers to access. Overall, patients who participated in televisits were satisfied with the experience and felt that the care met their medical needs; however, those who were older were more likely to experience technical difficulties and prefer in-person visits, while those with less education were less likely to feel that their questions were addressed in an understandable way.When thoughtfully designed, telemedicine practices can be an effective model for patient care, though implementation must consider population characteristics including age, education, and socioeconomic status, and strategies such as ease of access versus optimization of communication strategies should be tailored to meet individual patient needs.

How Do Programs Measure Resident Performance? A Multi-Institutional Inventory of General Surgery Assessments

OBJECTIVE

To perform an inventory of assessment tools in use at surgical residency programs and their alignment with the Milestone Competencies.

DESIGN

We conducted an inventory of all assessment tools from a sample of general surgery training programs participating in a multi-center study of resident operative development in the United States. Each instrument was categorized using a data extraction tool designed to identify criteria for effective assessment in competency based education and according to which Milestone Competency was being evaluated. Tabulations of each category were then analyzed using descriptive statistics. Interviews with program directors and assessment coordinators were conducted to understand each instrument's intended use within each program.

SETTING

Multi-institutional review of general surgery assessment programs.

PARTICIPANTS

We identified assessment tools used by 10 general surgery programs during the 2019 to 2020 academic year. Programs were selected from a cohort already participating in a separate research study of resident operative development in the United States.

RESULTS

We identified 42 unique assessment tools used. Each program used an average of 7.2 (range 4-13) unique assessment instruments to measure performance, of which only 5 (11.9%) were used by at least 1 other program in our sample. Of all assessments, 59.5% were used monthly or less frequently. The majority (66.7%) of instruments were retrospective global assessments, rather than discrete observed performances. There were 4 (9.5%) instruments with established reliability or validity evidence. Across programs there was also significant variation in the volume of assessment used to evaluate residents, with the median total number of evaluations/trainee across all Milestone Competencies being 217 (IQR 78) per year. Patient care was the most frequently evaluated Milestone Competency.

CONCLUSIONS

General surgical assessment systems predominantly employ non-standardized global assessment tools that lack reliability or validity evidence. This variability makes it challenging to interpret and compare competency standards across programs. A standardized assessment toolkit with established reliability and validity evidence would allow training programs to measure the competence of their trainees more uniformly and understand where improvements in our training system can be made.

Current Milestones Assessment Practices, Needs, and Challenges of Program Directors: A Collective Case Study in a Pediatric Hospital Setting

Introduction

Accreditation Council for Graduate Medical Education's (ACGME's) Milestones assessment requirement has placed new demands on Program Directors (PDs), especially those with limited knowledge of assessment and evaluation activities. There is a lack of clarity on how Program Director (PDs)/Associate PDs (APDs) are effectively implementing milestones assessment and evaluation practices in the Graduate Medical Education programs. The purpose of this study was to investigate current assessment practices, needs, and challenges of PDs in implementing milestones assessment within their residency and fellowship programs in a pediatric hospital setting.

Methods

This study used a collective case study approach to obtain information from PDs, APDs, and Clinical Competency Committee (CCC) Chairs in 19 graduate programs at a pediatric hospital. We used structured meetings with planned agendas and a pre-formatted template to itemize program needs/difficulties/challenges in the milestone assessment. We used cross-case thematic content anal­ysis to identify categories and themes to compare differences and commonalities across programs.

Results

A total of 38 PDs, APDs, and CCC Chairs from 19 different specialties/subspe­cialties participated in this study. Thirteen types of assessment and evaluation tools were consistently used across programs. Three categories emerged in relation to those assessment and evaluation types (direct, indirect, and multi-source). Rotation evaluation (84.2%), direct observation (73.2%), and 360-degree assessment (68.4%) were primarily used for measuring patient care among the six core competencies. Programs’ needs varied from curriculum and assessment tool development to alignment of milestones items, and to creating a sys­tematic assessment management plan. The most common challenges were difficulties related to logistics and tracking of evaluation in the survey management system (52.6%), challenges with time management (47.3%), and difficulty in determining and interpret­ing the milestones’ numbers and levels (31.5%).

Conclusions

Milestones assessment and evaluation in medical education can be a challenge, but a priority for many training programs. Our study indicated that milestones assessment and evaluation in medical education are far more com­plex than we expect. Multiple assessment methods must be utilized to evaluate all essential competencies for accurate measurement of trainees’ performance abilities. Our study uncovered several issues PDs faced during the implementation of milestones assessment and needs and challenges.

Professionalism and inter-communication skills (ICS): a multi-site validity study assessing proficiency in core competencies and milestones in medical learners

BACKGROUND

Interpersonal and Communication Skills (ICS) and Professionalism milestones are challenging to evaluate during medical training. Paucity in proficiency, direction and validity evidence of assessment tools of these milestones warrants further research. We validated the reliability of the previously-piloted Instrument for Communication skills and Professionalism Assessment (InCoPrA) in medical learners.

METHODS

This validity approach was guided by the rigorous Kane's Framework. Faculty-raters and standardized patients (SPs) used their respective InCoPrA sub-component to assess distinctive domains pertinent to ICS and Professionalism through multiple expert-built simulated-scenarios comparable to usual care. Evaluations included; inter-rater reliability of the faculty total score; the correlation between the total score by the SPs; and the average of the total score by two-faculty members. Participants were surveyed regarding acceptability, realism, and applicability of this experience.

RESULTS

Eighty trainees and 25 faculty-raters from five medical residency training sites participated. ICC of the total score between faculty-raters was generally moderate (ICC range 0.44-0.58). There was on average a moderate linear relationship between the SPs and faculty total scores (Pearson correlations range 0.23-0.44). Majority of participants ascertained receiving a meaningful, immediate, and comprehensive patient-faculty feedback.

CONCLUSIONS

This work substantiated that InCoPrA was a reliable, standardized, evidence-based, and user-friendly assessment tool for ICS and Professionalism milestones. Validating InCoPrA showed generally-moderate agreeability and high acceptability. Using InCoPrA also promoted engaging all stakeholders in medical education and training-faculty, learners, and SPs-using simulation-media as pathway for comprehensive feedback of milestones growth.

Relationship Between "Simulated Patient Scenarios and Role-Playing" Method and OSCE Performance in Senior Anesthesiology Residents: A Correlation Assessment Study

BACKGROUND

"Simulated-patient scenarios and role-playing" and OSCE are among the many non-traditional education methods with variable results in different clinical settings.

OBJECTIVES

This cross-sectional study was performed to assess the correlation between the results of these two methods in senior anesthesiology residents, with a special focus on four of the six ACGME core competencies.

METHODS

During two years, senior anesthesiology residents were subject to "simulated patient scenario and role-playing" sessions. Two faculty members took the role of the patient and one of the relatives. An objective checklist with 15 items was prepared to be rated by other department faculty members. Meanwhile, an ordered pattern of OSCE was prepared to cover four core competencies that were more related to this academic process (from a total of six core competencies). The mean and standard deviation of the score of each of the 15 items in the checklist were calculated. The correlation between cumulative checklist scoring results and OSCE exam results was assessed. A P value of less than 0.05 was considered significant.

RESULTS

A total of 40 senior anesthesiology residents, with 344 assessments by faculty members in 40 sessions, were enrolled in the study. The questionnaire's Cronbach's alpha reliability was 0.74. No statistically significant disparity was detected between the results of the two assessment methods, while the results of the two assessments had a significant correlation (two-tailed correlation coefficient = 0.886; P value < 0.001).

CONCLUSIONS

There was an objective relationship between the results of "simulated patient scenario and role-playing" strategies and the results of OSCE exams using an observer-based rating method. Thus, they could be used as surrogates in the assessment of core clinical competencies of senior anesthesiology residents.

Improving plastic surgery resident education and quality of care with outcomes feedback using the surgery report card: An initial experience

BACKGROUND

The practice of tracking and analyzing surgical outcomes is essential to becoming better surgeons. However, this feedback system is largely absent in residency training programs. Thus, we developed a Surgery Report Card (SRC) for residents performing tissue expander (TE)-based breast reconstruction and report our initial experience with its implementation.

METHODS

We performed a systematic review and meta-analysis for TE-based breast reconstructions and compared outcomes to our retrospective cohort. The primary outcome was overall complications. The SRC compares patient and complication statistics for resident-led teams to the meta-analysis.

RESULTS

The meta-analysis included 12 studies, with 2093 patients (2982 breasts) that underwent TE-based reconstruction. The pooled complication rate was 26.9%; infection was most common (8.3%); failure rate was 5.9%. Our cohort included 144 patients (245 breasts) among 13 resident-led teams. Overall complication rate was 31.8%; infections were most frequent (17.6%) and failure rate was 7.3%. Our cohort had significantly higher BMIs (29.7 vs 25.4, p<0.0001) more diabetics (6.9% vs. 3.2%, p = 0.02), and more patients receiving adjuvant radiation therapy (41.4 vs 26.3%, p<0.0001). Every 3 months, residents receive a customized SRC of their cases, with the meta-analysis used as a benchmark. A survey demonstrated the SRC made residents reconsider surgical technique and more conscientious surgeons, and would like it implemented for other procedures during residency.

CONCLUSIONS

The implementation of our SRC for TE-based breast reconstruction establishes a baseline for surgical performance comparison for residents, demonstrates that residents can safely perform the procedure, and allows for critiquing of surgical techniques to improve patient care.

Assessing medical student performance of Entrustable Professional Activities: A mixed methods comparison of Co-Activity and Supervisory Scales

Introduction: Observations of medical student participation in entrustable professional activities (EPAs) provide insight into the student's ability to synthesize competencies across domains and effectively function in different clinical scenarios. Both Supervisory and Co-Activity Assessment Scales have been recommended for use with medical students.Methods: Students were assessed on EPAs during Acting Internships in Medicine and Pediatrics. Two rating scales were modified based on expert review and included throughout the 2017-18 academic year. Statistical analysis was conducted to clarify relationships between the scales. Raters were interviewed to explore their interpretations and response processes.Results: The results of the McNemar test suggest that the scales are different (p-value <.01). Co-activity and Supervisory EPA ratings are related, but not interchangeable. This finding concurs with themes that emerged from response process interviews: (1) the scales are not directly parallel (2) rater preference depends on diverse factors and (3) rater comments are crucial for guiding students' future learning.Conclusion: The modified Chen Supervisory Scale and the modified Ottawa Co-Activity Scales are measuring different aspects of the entrustable professional activity landscape. Both scales can provide useful information to the learner and the assessment system, but they should not be treated as interchangeable assessments.