Current State of the Medical Student Performance Evaluation: A Tool for Reflection for Residency Programs

Variation in Transcript Reports Among Residency Applicants: An Anesthesia Program's Perspective

Background

With recent changes made to move USMLE Step 1 and COMLEX Level 1 scores to Pass/Fail, it becomes necessary to find other metrics to evaluate residency candidates. One conserved metric included in all residency applications is medical school transcripts. This study aims to highlight the highly varied transcript reporting in a new era of holistic applicant review.

Methods

Medical school transcripts were extracted from the Electronic Residency Application Service applications to our anesthesiology residency program for the 2021-2022 application cycle. All personally identifiable information was removed. Results were categorized and tallied by 2 independent reviewers. Overall, we assessed transcript information from 156 allopathic and osteopathic medical schools. Transcript data were separated into 9 different categories.

Results

The most common grading system for allopathic medical schools was Pass/Fail. The most common grading system for osteopathic medical schools was Pass/Fail and Letter Grades. There were several medical schools that had unique grading systems and many of those did not provide a grading key for interpretation. Less than half of the allopathic and osteopathic schools offered Honors or High Pass in their grading systems, often with little information provided as to how these grades were earned.

Conclusion

The information provided on medical school transcripts is extremely variable. Although many schools reported grades as Pass/Fail, there was no majority or consistent presentation among the transcripts. Much of the information provided on transcripts required interpretation by its reviewer and made the process of holistic applicant review more difficult.

A Systematic Review of Metrics Utilized in the Selection and Prediction of Future Performance of Residents in the United States

Background Aligning resident and training program attributes is critical. Many programs screen and select residents using assessment tools not grounded in available evidence. This can introduce bias and inappropriate trainee recruitment. Prior reviews of this literature did not include the important lens of diversity, equity, and inclusion (DEI). Objective This study's objective is to summarize the evidence linking elements in the Electronic Residency Application Service (ERAS) application with selection and training outcomes, including DEI factors. Methods A systematic review was conducted on March 30, 2022, concordant with PRISMA guidelines, to identify the data supporting the use of elements contained in ERAS and interviews for residency training programs in the United States. Studies were coded into the topics of research, awards, United States Medical Licensing Examination (USMLE) scores, personal statement, letters of recommendation, medical school transcripts, work and volunteer experiences, medical school demographics, DEI, and presence of additional degrees, as well as the interview. Results The 2599 identified unique studies were reviewed by 2 authors with conflicts adjudicated by a third. Ultimately, 231 meeting inclusion criteria were included (kappa=0.53). Conclusions Based on the studies reviewed, low-quality research supports use of the interview, Medical Student Performance Evaluation, personal statement, research productivity, prior experience, and letters of recommendation in resident selection, while USMLE scores, grades, national ranking, attainment of additional degrees, and receipt of awards should have a limited role in this process.

A Retrospective Analysis of Medical Student Performance Evaluations, 2014-2020: Recommend with Reservations

BACKGROUND

The Medical Student Performance Evaluations (MSPE) is a cornerstone of residency applications. Little is known regarding adherence to Association of American Medical Colleges (AAMC) MSPE recommendations and longitudinal changes in MSPE content.

OBJECTIVES

Evaluate current MSPE quality and longitudinal changes in MSPE and grading practices.

DESIGN

Retrospective analysis.

PARTICIPANTS

Students from all Liaison Committee on Medical Education (LCME)-accredited medical schools from which the Stanford University Internal Medicine residency program received applications between 2014-2015 and 2019-2020.

MAIN MEASURES

Inclusion of key words to describe applicant performance and metrics thereof, including distribution among students and key word assignment explanation; inclusion of clerkship grades, grade distributions, and grade composition; and evidence of grade inflation over time.

KEY RESULTS

MSPE comprehensiveness varied substantially among the 149 schools analyzed. In total, 25% of schools provided complete information consistent with AAMC recommendations regarding key word/categorization of medical students and clerkship grades in 2019-2020. Seventy-seven distinct key word terms appeared across the 139 schools examined in 2019-2020. Grading practices markedly varied, with 2-83% of students receiving the top internal medicine clerkship grade depending on the year and school. Individual schools frequently changed key word and grading practices, with 33% and 18% of schools starting and/or stopping use of key words and grades, respectively. Significant grade inflation occurred over the 6-year study period, with an average 14% relative increase in the proportion of students receiving top clerkship grades.

CONCLUSIONS

A minority of schools complies with AAMC MSPE guidelines, and MSPEs are inconsistent across time and schools. These practices may impair evaluation of students within and between schools.

Does Objective Scoring of Applications for Radiology Residency Affect Diversity?

RATIONALE AND OBJECTIVES

It is important to recognize if reliance on certain factors in applications affects selection of trainees from under-represented groups. Our purpose is to determine if objective scoring of radiology residency applications based on quantifiable data regarding academic performance, United States Medical Licensing Examination (USMLE) scores and research productivity affects selection of female and under-represented minority (URM) candidates.

MATERIALS AND METHODS

We reviewed 502 applications from three successive match cycles from United States allopathic medical students. Each application was scored for academic performance, USMLE results and research productivity determining an overall score. The scores of males were compared to females and URM were compared to non-URM candidates. USMLE cutoff scores were evaluated for disparate effects.

RESULTS

There were 348 male, 154 female, 73 URM and 429 non-URM candidates. For male versus female applicants, there was no significant difference in mean academic performance, USMLE Step 2 Clinical Knowledge and research productivity scores. Males had higher mean USMLE Step 1 (p = 0.005) and overall candidate scores (p = 0.02). Between URM and non-URM candidates there was no significant difference in academic performance. Non-URM applicants had higher mean USMLE Step 1 (p = 0.008), USMLE Step 2 (p = 0.002), research productivity (p = 0.001) and overall scores (p = 0.02). Use of USMLE cutoff scores demonstrated disparate effects on female and URM candidates.

CONCLUSION

Objective scoring of applications and use of USMLE cutoff scores can disadvantage candidates from underrepresented groups. Screening filters can affect the diversity of candidate pools for radiology residency.