Standardization in the MSPE: Key Tensions for Learners, Schools, and Residency Programs

Dynamic Opportunities for Medical Students to Assume the Roles of "Medical Teacher"

The traditional undergraduate medical education curriculum focuses on bolstering knowledge for practice and building clinical skills. However, as future clinicians, medical students will be tasked with teaching throughout their careers, first as residents and then as attendings. Here, we describe teaching opportunities for students that foster their development as future teachers and potential clinician educators. These offerings are diverse in their focus and duration and are offered across various levels of the curriculum - including course-based learning, longitudinal electives, and extra-curricular opportunities for medical students who have a passion for teaching.

Comparing Machine Learning Models and Human Raters When Ranking Medical Student Performance Evaluations

Background

The Medical Student Performance Evaluation (MSPE), a narrative summary of each student's academic and professional performance in US medical school is long, making it challenging for residency programs evaluating large numbers of applicants.

Objective

To create a rubric to assess MSPE narratives and to compare the ability of 3 commercially available machine learning models (MLMs) to rank MSPEs in order of positivity.

Methods

Thirty out of a possible 120 MSPEs from the University of Central Florida class of 2020 were de-identified and subjected to manual scoring and ranking by a pair of faculty members using a new rubric based on the Accreditation Council for Graduate Medical Education competencies, and to global sentiment analysis by the MLMs. Correlation analysis was used to assess reliability and agreement between student rank orders produced by faculty and MLMs.

Results

The intraclass correlation coefficient used to assess faculty interrater reliability was 0.864 (P<.001; 95% CI 0.715-0.935) for total rubric scores and ranged from 0.402 to 0.768 for isolated subscales; faculty rank orders were also highly correlated (rs=0.758; P<.001; 95% CI 0.539-0.881). The authors report good feasibility as the rubric was easy to use and added minimal time to reading MSPEs. The MLMs correctly reported a positive sentiment for all 30 MSPE narratives, but their rank orders produced no significant correlations between different MLMs, or when compared with faculty rankings.

Conclusions

The rubric for manual grading provided reliable overall scoring and ranking of MSPEs. The MLMs accurately detected positive sentiment in the MSPEs but were unable to provide reliable rank ordering.

A Retrospective Analysis of Medical Student Performance Evaluations, 2014-2020: Recommend with Reservations

BACKGROUND

The Medical Student Performance Evaluations (MSPE) is a cornerstone of residency applications. Little is known regarding adherence to Association of American Medical Colleges (AAMC) MSPE recommendations and longitudinal changes in MSPE content.

OBJECTIVES

Evaluate current MSPE quality and longitudinal changes in MSPE and grading practices.

DESIGN

Retrospective analysis.

PARTICIPANTS

Students from all Liaison Committee on Medical Education (LCME)-accredited medical schools from which the Stanford University Internal Medicine residency program received applications between 2014-2015 and 2019-2020.

MAIN MEASURES

Inclusion of key words to describe applicant performance and metrics thereof, including distribution among students and key word assignment explanation; inclusion of clerkship grades, grade distributions, and grade composition; and evidence of grade inflation over time.

KEY RESULTS

MSPE comprehensiveness varied substantially among the 149 schools analyzed. In total, 25% of schools provided complete information consistent with AAMC recommendations regarding key word/categorization of medical students and clerkship grades in 2019-2020. Seventy-seven distinct key word terms appeared across the 139 schools examined in 2019-2020. Grading practices markedly varied, with 2-83% of students receiving the top internal medicine clerkship grade depending on the year and school. Individual schools frequently changed key word and grading practices, with 33% and 18% of schools starting and/or stopping use of key words and grades, respectively. Significant grade inflation occurred over the 6-year study period, with an average 14% relative increase in the proportion of students receiving top clerkship grades.

CONCLUSIONS

A minority of schools complies with AAMC MSPE guidelines, and MSPEs are inconsistent across time and schools. These practices may impair evaluation of students within and between schools.

Competency-based medical education across the continuum: How well aligned are medical school EPAs to residency milestones?

INTRODUCTION

Competency-based medical education (CBME) provides a framework for describing learner progression throughout training. However, specific approaches to CBME implementation vary widely across educational settings. Alignment between various methods used across the continuum is critical to support transitions and assess learner performance. The purpose of this study was to investigate alignment between CBME frameworks used in undergraduate medical education (UME) and graduate medical education (GME) settings using the US context as a model.

METHOD

The authors analyzed content from the core entrustable professional activities for entering residency (Core EPAs; UME model) and residency milestones (GME model). From that analysis, they performed a series of cross-walk activities to investigate alignment between frameworks. After independent review, authors discussed findings until consensus was reached.

RESULTS

Some alignment was found for activities associated with history taking, physical examination, differential diagnosis, patient safety, and interprofessional care; however, there were far more examples of misalignment.

CONCLUSIONS

These findings highlight challenges creating alignment of assessment frameworks across the continuum of training. The importance of these findings includes implications for assessment and persistence of the educational gap across UME and GME. The authors provide four next steps to improve upon the continuum of education.

Moving toward Mastery: Changes in Student Perceptions of Clerkship Assessment with Pass/Fail Grading and Enhanced Feedback

ProblemClerkship grades contribute to a summative assessment culture in clerkships and can therefore interfere with students' learning. For example, by focusing on summative, tiered clerkship grades, students often discount accompanying feedback that could inform future learning. This case report seeks to explore whether an assessment system intervention which eliminated tiered grades and enhanced feedback was associated with changes in student perceptions of clerkship assessment and perceptions of the clinical learning environment. Intervention: In January 2019, our institution eliminated tiered clerkship grading (honors/pass/fail) for medical students during the core clerkship year and implemented pass/fail clerkship grading along with required twice weekly, work-based assessments for formative feedback. Context: In this single institution, cross-sectional survey study, we collected data from fourth-year medical students one year after an assessment system intervention. The intervention entailed changing from honors/pass/fail to pass/fail grading in all eight core clerkships and implementing an electronic system to record twice-weekly real-time formative work-based assessments. The survey queried student perceptions on the fairness and accuracy of grading and the clinical learning environment-including whether clerkships were mastery- or performance-oriented. We compared responses from students one year after the assessment intervention to those from the class one year before the intervention. Comparisons were made using unpaired, two-tailed t-tests or chi-squared tests as appropriate with Cohen's d for effect size estimation for score differences. Content analysis was used to analyze responses from two open-ended questions about feedback and grading. Impact: Survey response rates were similar before and after intervention (76% (127/168) vs. 72% (118/163), respectively) with no between-group differences in demographics. The after-intervention group showed statistically significant increases in the following factors: "grades are transparent and fair" (Cohen's d = 0.80), "students receive useful feedback" (d = 0.51), and "resident evaluation procedures are fair" (d = 0.40). After-intervention respondents perceived the clerkship learning environment to be more mastery-oriented (d = 0.52), less performance approach-oriented (d = 0.63), and less performance avoid-oriented (d = 0.49). There were no statistical differences in the factors "attending evaluation procedures are fair," "evaluations are accurate," "evaluations are biased," or "perception of stereotype threat." Open-ended questions revealed student recommendations to improve clerkship summary narratives, burden of work-based assessment, and in-person feedback. Lessons Learned: After an assessment system change to pass/fail grading with work-based assessments, we observed moderate to large improvements in student perceptions of clerkship grading and the mastery orientation of the learning environment. Our intervention did not improve perceptions around bias in assessment in clerkships. Other medical schools may consider similar interventions to begin to address student concerns with clerkship assessment and promote a more adaptive learning environment.

Review of the Medical Student Performance Evaluation: analysis of the end-users' perspective across the specialties

The Medical Student Performance Evaluation (MSPE) is an important tool of communication used by program directors to make decisions in the residency application process. To understand the perspective and usage of the MSPE across multiple medical specialties now and in anticipation of the planned changes in USMLE Step 1 score-reporting. A survey instrument including quantitative and qualitative measures was developed and piloted. The final survey was distributed to residency programs across 28 specialties in 2020 via the main contact on the ACGME listserv. Of the 28 specialties surveyed, at least one response was received from 26 (93%). Eight percent of all programs (364/4675) responded to the survey, with most respondents being program directors. Usage of the MSPE varied among specialties. Approximately 1/3 of end-users stated that the MSPE is very or extremely influential in their initial screening process. Slightly less than half agreed or strongly agreed that they trust the information to be an accurate representation of applicants, though slightly more than half agree that the MSPE will become more influential once USMLE Step 1 becomes pass/fail. Professionalism was rated as the most important component and noteworthy characteristics among the least important in the decision-making process. Performance in the internal medicine clerkship was rated as the most influential while neurology and psychiatry performances were rated as less influential. Overwhelmingly, respondents suggested that including comparative performance and/or class rank would make the MSPE more useful once USMLE Step 1 becomes pass/fail. MSPE end-users across a variety of specialties utilize this complex document in different ways and value it differentially in their decision-making processes. Despite this, continued mistrust of the MSPE persists. A better understanding of end-users' perceptions of the MSPE offers the UME community an opportunity to transform the MSPE into a highly valued, trusted document of communication.

Underrepresented in medicine in graduate medical education: Historical trends, bias, and recruitment practices

Health inequities disproportionately impact children of color in the United States. Research demonstrates that a diverse physician workforce that reflects the demographic profile of the United States population has been associated with better health outcomes and physicians of color are more likely to work in communities that are medically underserved. Therefore, a key driver to reducing health inequities is recruiting a physician workforce that reflects the racial/ethnic composition of the population. Although efforts on increasing UIM applicants and matriculants into medical school and more recently, graduate medical education (GME) programs have shown some small successes, there has still not been significant change in decades. This narrative review examined the historical trends and current landscape of recruitment in GME to better understand the barriers and facilitators that continue to exist and identify successful interventions that might serve as a model to support recruitment of UIM applicants. There is evidence for racial disparities and bias in traditional residency selection metrics (USMLE, Clerkship Evaluations, MSPE, AOA status, etc.) and a need for more holistic recruitment strategies. Along with implementing new holistic recruitment practices, there is a need for ongoing research and collaboration to study inclusive and equitable learning environments and assessments that will eliminate racial disparities in educational outcomes. Additionally, there must be better longitudinal data utilization and transparency across Electronic Residency Application Services (ERAS) and National Resident Matching Program (NRMP) for accountability and tracking of interventions.

Current State of the Medical Student Performance Evaluation: A Tool for Reflection for Residency Programs

BACKGROUND

The Medical Student Performance Evaluation (MSPE) provides important information to residency programs. Despite recent recommendations for standardization, it is not clear how much variation exists in MSPE content among schools.

OBJECTIVES

We describe the current section content of the MSPE in US allopathic medical schools, with a particular focus on variations in the presentation of student performance.

METHODS

A representative MSPE was obtained from 95.3% (143 of 150) of allopathic US medical schools through residency applications to the Zucker School of Medicine at Hofstra/Northwell in select programs for the 2019-2020 academic year. A manual data abstraction tool was piloted in 2018-2019. After training, it was used to code all portions of the MSPE in this study. The results were analyzed, and descriptive statistics were reported.

RESULTS

In preclinical years, 30.8% of MSPEs reported data regarding performance of students beyond achieving "passes" in a pass/fail curriculum. Only half referenced performance in the fourth year including electives, acting internships, or both. About two-thirds of schools included an overall descriptor of comparative performance in the final paragraph. Among these schools, a majority provided adjectives such as "outstanding/excellent/very good/good," while one-quarter reported numerical data categories. Regarding clerkship grades, there were numerous nomenclature systems used.

CONCLUSIONS

This analysis demonstrates the existence of extreme variability in the content of MSPEs submitted by US allopathic medical schools in the 2019-2020 cycle, including the components and nomenclature of grades and descriptors of comparative performance, display of data, and inclusion of data across all years of the medical education program.

Medical School to Residency: How Can We Trust the Process?

Background

To say that the transition from undergraduate medical education (UME) to graduate medical education (GME) is under scrutiny would be an understatement. Findings from a panel discussion at the 2018 Association of American Medical Colleges Annual meeting entitled, “Pass-Fail in Medical School and the Residency Application Process and Graduate Medical Education Transition” addressed what and when information should be shared with residency programs, and how and when that information should be shared.

Materials and Methods

Over 250 participants representing UME and GME (e.g. leadership, faculty, medical students) completed worksheets addressing these questions. During report-back times, verbal comments were transcribed in real time, and written comments on worksheets were later transcribed. All comments were anonymous. Thematic analysis was conducted manually by the research team to analyze the worksheet responses and report back comments.

Results

Themes based on suggestions of what information should be shared included the following: 1) developmental/assessment benchmarks such as demonstrating the ability/competencies to do clinical work; 2) performance on examinations; 3) grades and class ranking; 4) 360 evaluations; 5) narrative evaluations; 6) failures/remediation/gaps in training; 7) professionalism lapses; 8) characteristics of students such as resiliency/reliability; and 9) service/leadership/participation. In terms of how this information should be shared, the participants suggested enhancements to the current process of transmitting documents rather than alternative methods (e.g., video, telephonic, face-to-face discussions) and information sharing at both the time of the match and again near/at graduation to include information about post-match rotations.

Discussion

Considerations to address concerns with the transition from medical school to residency include further enhancements to the Medical Student Performance Evaluation, viewing departmental letters as ones of evaluation and not recommendation, a more meaningful educational handoff, and limits on the number of residency applications allowed for each student. The current medical education environment is ready for meaningful change in the UME to GME transition.

Innovations in Geroscience to enhance mobility in older adults

Aging is the primary risk factor for functional decline; thus, understanding and preventing disability among older adults has emerged as an important public health challenge of the 21st century. The science of gerontology - or geroscience - has the practical purpose of "adding life to the years." The overall goal of geroscience is to increase healthspan, which refers to extending the portion of the lifespan in which the individual experiences enjoyment, satisfaction, and wellness. An important facet of this goal is preserving mobility, defined as the ability to move independently. Despite this clear purpose, this has proven to be a challenging endeavor as mobility and function in later life are influenced by a complex interaction of factors across multiple domains. Moreover, findings over the past decade have highlighted the complexity of walking and how targeting multiple systems, including the brain and sensory organs, as well as the environment in which a person lives, can have a dramatic effect on an older person's mobility and function. For these reasons, behavioral interventions that incorporate complex walking tasks and other activities of daily living appear to be especially helpful for improving mobility function. Other pharmaceutical interventions, such as oxytocin, and complementary and alternative interventions, such as massage therapy, may enhance physical function both through direct effects on biological mechanisms related to mobility, as well as indirectly through modulation of cognitive and socioemotional processes. Thus, the purpose of the present review is to describe evolving interventional approaches to enhance mobility and maintain healthspan in the growing population of older adults in the United States and countries throughout the world. Such interventions are likely to be greatly assisted by technological advances and the widespread adoption of virtual communications during and after the COVID-19 era.

Pass/Fail Score Reporting for USMLE Step 1: An Opportunity to Redefine the Transition to Residency Together

The upcoming transition of the United States Medical Licensing Examination Step 1 to pass/fail score reporting has wide-ranging implications for the medical education community. The decision to discontinue 3-digit numeric score reporting comes following advocacy to change the exam because of its disproportionate importance in the residency match process, the negative impact the exam has on student well-being, and the prominent influence the exam has on medical school curricula. Shifting to pass/fail score reporting for Step 1 creates significant uncertainties to be addressed by the stakeholders in the transition from undergraduate medical education to graduate medical education. In this Invited Commentary, the authors reflect on the positive implications of this scoring change, potential negative ramifications that need to be proactively addressed, and future ways that the transition to residency can be optimized. The transition to pass/fail score reporting for Step 1 creates an opportunity to redefine the residency match process, to support applicant decision making, and to encourage a holistic review of applicants. These changes to the review process will require grace and trust among stakeholders, including students, as well as active support for students who might be negatively impacted during a complex implementation phase. By removing the dominance of Step 1 numeric scores from the residency selection process, the change to pass/fail scoring provides a unique opportunity for all stakeholders to work together and redefine the transition to residency while protecting students from unintended negative consequences.