The Lack of Standardization of Allopathic and Osteopathic Medical School Grading Systems and Transcripts

Are Pre-clerkship Remediation, Grading, and Reporting Practices Equitable in the U.S.? A National Survey

Phenomenon: With the proliferation of pass/fail grading practices in the pre-clerkship phase of undergraduate medical education, questions arise about the transparency and variability of grading and grade reporting practices, raising issues of equity in assessment, particularly regarding residency matching. The purpose of this survey was to determine the remediation and academic performance reporting practices of United States (U.S.) allopathic medical schools in the pre-clerkship phase of their curricula. Approach: After an extensive literature search and feedback from curriculum deans and learning experts, we developed a survey that we sent in the Spring of 2022 to pre-clerkship curriculum officials at all 154 accredited U.S. allopathic medical schools. It addressed curriculum content and structure; pre-clerkship remediation (e.g., course retakes) and reporting (e.g., permanency of transcript notation) practices; documentation and reporting of nonacademic competencies; and participant opinions and recommendations regarding reporting, transparency, and equity. We generated descriptive statistics and did manifest coding of open-ended responses. Findings: We had a response rate of 40% (62/155), with over 71% indicating mainly organ systems-based curricula. Depending on the situation, there were a wide range of remediation approaches for single- and multiple-course failures, including tutoring or learning support, re-exams, and referrals to a promotion board. Professionalism concerns were a top priority to report to residency directors, with significant variability in respondent opinions and practices in reporting remedial activities. Respondents were concerned about equity, both in terms of flexible grading practices and transparency of reporting practices. Insights: The variability in reporting practices across schools, while allowing holistic and individualized approaches to academic support, also creates potential inequities. More work is needed to understand how different reporting practices across institutions may disadvantage marginalized and minoritized student groups at different points in their preparation.

Analysis of variability and trends in medical school clerkship grades

Background

Medical school clerkship grades are used to evaluate orthopedic surgery residency applicants, however, high interinstitutional variability in grade distribution calls into question the utility of clerkship grades when evaluating applicants from different medical schools. This study aims to evaluate the variability in grade distribution among medical schools and look for trends in grade distribution over recent years.

Methods

Applications submitted to Rush University's orthopedic surgery residency program from 2015, 2019, and 2022 were collected from the Electronic Residency Application Service. Applications from the top 100 schools according to the 2023-2024 U.S. News and World Report Research Rankings were reviewed. The percentage of "honors" grades awarded by medical schools for the surgery and internal medicine clerkships were extracted from applicants' Medical Student Performance Evaluation letters.

Results

The median percentage of honors given in 2022 was 36.0 % (range 10.0-82.0) for the surgery clerkship and 33.0 % (range 6.7-80.0) for the internal medicine clerkship. Honors were given 6.6 % more in the surgery clerkship in 2022 compared to 2015. There was a negative correlation between a higher (worse) U.S. News and World Report research ranking and the percentage of honors awarded in 2022 for the surgery and internal medicine clerkships.

Conclusion

There is substantial interinstitutional variability in the rate that medical schools award an "honors" grade with evidence of grade inflation in the surgery clerkship. Residency programs using clerkship grades to compare applicants should do so cautiously provided the variability demonstrated in this study.

An analysis of osteopathic medical students applying to surgical residencies following transition to a single graduate medical education accreditation system

CONTEXT

Upon requests from osteopathic medical schools, the National Resident Matching Program (NRMP) Charting Outcomes were redesigned to include osteopathic medical school seniors beginning in 2018 and one joint graduate medical education (GME) accreditation system, the Accreditation Council for Graduate Medical Education (ACGME), formed in 2020.

OBJECTIVES

The goal of this study is to analyze the match outcomes and characteristics of osteopathic applicants applying to surgical specialties following the ACGME transition.

METHODS

A retrospective analysis of osteopathic senior match outcomes in surgical specialties from the NRMP Main Residency Match data from 2020 to 2022 and the NRMP Charting Outcomes data from 2020 to 2022 was performed.

RESULTS

For surgical specialties, results show matching increased as United States Medical Licensing Examination (USMLE) Step 2 CK (clinical knowledge) and Comprehensive Osteopathic Medical Licensing Examination (COMLEX) Level 2 CE (cognitive evaluation) scores increased along with the number of contiguous rankings (p<0.001). The greatest indication for matching looking at scores alone were those who scored greater than 230 on Step 2 CK compared to below (p<0.001) and above 650 on Level 2 CE (p<0.001). However, those who scored 240 (p=0.025) on Step 2 CK were just as likely to match as those who scored 250 (p=0.022) when compared to those who scored below those scores. Increasing research involvement had little to no significance with the likelihood of matching across most surgical subspecialties.

CONCLUSIONS

Our study demonstrates that there are unique thresholds for Step 2 CK scores, Level 2 CE scores, and the number of contiguous ranks for each surgical specialty that, when reached, are significantly associated with match success. Although certain board score delineations are linked with higher match success rates, the rates level off after this point for most surgical specialties and do not significantly increase further with higher scores. In addition, thresholds within contiguous ranks for increasing match likelihood exist and vary across surgical specialties. Overall, this study highlights that the quantitative metrics utilized to assess applicants lack the correlation reported historically, and the data presently available need to be more substantiated.

Medical School Clinical Knowledge Exam Scores, Not Demographic or Other Factors, Associated With Residency In-Training Exam Performance

BACKGROUND

The anesthesiology in-training exam (ITE) is a 200-item multiple-choice assessment completed annually by physician residents. Because all matriculated U.S. Department of Defense (DoD) anesthesiology residents are "hired" by the DoD after residency graduation, it is important to ensure that ITE performance, as a proxy for core competencies achievement, is maximized.

METHODS

Graduated resident program files from 2013 to 2020 were queried for age, sex, matriculant status (medical student vs. other), medical school (Uniformed Services University vs. other), military service (Army vs. Air Force), preresidency military service (yes vs. no), U.S. Medical Licensing Exam (USMLE) Step 2 Clinical Knowledge (CK) score, and the American Board of Anesthesiologists ITE Score from the third clinical anesthesia year (CA-3 year).

RESULTS

For every 1-point increase in USMLE Step 2 CK true z-score, the CA-3 ITE z-score increased by 0.59 points. Age was not associated with CA-3 ITE z-score in any dataset regression. Categorical covariates of sex, application status, medical school, service, and preresidency military service were not significantly associated with CA-3 ITE z-score (all P >.05), as shown by estimated adjusted marginal means. The estimated adjusted grand mean of CA-3 ITE z-scores was 0.48 (standard error ± 0.14).

CONCLUSION

Resident physicians enter residency with varying degrees of past academic success, and it is important to develop early strategies to support them in acquiring the requisite knowledge base.

Review of the Medical Student Performance Evaluation: analysis of the end-users' perspective across the specialties

The Medical Student Performance Evaluation (MSPE) is an important tool of communication used by program directors to make decisions in the residency application process. To understand the perspective and usage of the MSPE across multiple medical specialties now and in anticipation of the planned changes in USMLE Step 1 score-reporting. A survey instrument including quantitative and qualitative measures was developed and piloted. The final survey was distributed to residency programs across 28 specialties in 2020 via the main contact on the ACGME listserv. Of the 28 specialties surveyed, at least one response was received from 26 (93%). Eight percent of all programs (364/4675) responded to the survey, with most respondents being program directors. Usage of the MSPE varied among specialties. Approximately 1/3 of end-users stated that the MSPE is very or extremely influential in their initial screening process. Slightly less than half agreed or strongly agreed that they trust the information to be an accurate representation of applicants, though slightly more than half agree that the MSPE will become more influential once USMLE Step 1 becomes pass/fail. Professionalism was rated as the most important component and noteworthy characteristics among the least important in the decision-making process. Performance in the internal medicine clerkship was rated as the most influential while neurology and psychiatry performances were rated as less influential. Overwhelmingly, respondents suggested that including comparative performance and/or class rank would make the MSPE more useful once USMLE Step 1 becomes pass/fail. MSPE end-users across a variety of specialties utilize this complex document in different ways and value it differentially in their decision-making processes. Despite this, continued mistrust of the MSPE persists. A better understanding of end-users' perceptions of the MSPE offers the UME community an opportunity to transform the MSPE into a highly valued, trusted document of communication.

The Effects of a Pass-Fail Curriculum on a Physician Assistant Program

PURPOSE

The grading scale for students in a physician assistant program of study is not standardized. Students may be evaluated on a traditional 5-tiered A to F scale or a pass-fail system. The decision to change from ordered grading to pass-fail at an established program in the southeast was done following a change in the affiliated School of Medicine. The purpose of this study was to review effects on student scores following such a change.

METHODS

The Physician Assistant National Certifying Exam (PANCE) and PACKRAT 2 exam scores for the last 2 cohorts of students scored in the 5-tiered system (2016, N = 60 and 2017, N = 59) were compared against the same for the first 2 cohorts (2018, N = 59 and 2019, N = 58) of the pass-fail system. Nonrandom sampling of all students in each cohort year was evaluated using 2-tailed t-testing.

RESULTS

A total of 236 student scores were evaluated using a 95% confidence interval. The traditionally scored classes outperformed all pass-fail cohorts (means 460.67/491.86 versus 503.34/493.92). P values were found to be significant at all values between the 5-tier scored classes and the pass-fail cohorts in PANCE scoring, resulting in failure to reject the null hypothesis. This was also true for the PACKRAT 2 with the exception of the 2019 cohort, which was significant only for outperformance of the other pass-fail cohort. For the purpose of this study, the only analysis performed was scoring.

CONCLUSION

For cohorts undergoing curricular change, unforeseen impacts on initial standardized exam scores may occur. In this study, PANCE scores for the first year of the 2 pass-fail cohorts decreased while the overall program scores remained at or above the national average. The pass-fail cohort did show an upward trend in the second year of the curriculum, suggesting that as programs become more familiar with the pass-fail system, steady improvements occur. This suggests that while an anticipated drop in initial scores may be expected, further studies are needed to evaluate the impact on stress reduction, long retention, and intraclass competition.

Current State of the Medical Student Performance Evaluation: A Tool for Reflection for Residency Programs

BACKGROUND

The Medical Student Performance Evaluation (MSPE) provides important information to residency programs. Despite recent recommendations for standardization, it is not clear how much variation exists in MSPE content among schools.

OBJECTIVES

We describe the current section content of the MSPE in US allopathic medical schools, with a particular focus on variations in the presentation of student performance.

METHODS

A representative MSPE was obtained from 95.3% (143 of 150) of allopathic US medical schools through residency applications to the Zucker School of Medicine at Hofstra/Northwell in select programs for the 2019-2020 academic year. A manual data abstraction tool was piloted in 2018-2019. After training, it was used to code all portions of the MSPE in this study. The results were analyzed, and descriptive statistics were reported.

RESULTS

In preclinical years, 30.8% of MSPEs reported data regarding performance of students beyond achieving "passes" in a pass/fail curriculum. Only half referenced performance in the fourth year including electives, acting internships, or both. About two-thirds of schools included an overall descriptor of comparative performance in the final paragraph. Among these schools, a majority provided adjectives such as "outstanding/excellent/very good/good," while one-quarter reported numerical data categories. Regarding clerkship grades, there were numerous nomenclature systems used.

CONCLUSIONS

This analysis demonstrates the existence of extreme variability in the content of MSPEs submitted by US allopathic medical schools in the 2019-2020 cycle, including the components and nomenclature of grades and descriptors of comparative performance, display of data, and inclusion of data across all years of the medical education program.

From Grading to Assessment for Learning: A Qualitative Study of Student Perceptions Surrounding Elimination of Core Clerkship Grades and Enhanced Formative Feedback

Problem: Medical students perceive honors grading during core clerkships as unfair and inequitable, and negatively impacting their learning and wellbeing. Eliminating honors grading, a powerful extrinsic motivator, and emphasizing formative feedback may address these problems and promote intrinsic motivation and learning. However, it is unknown how transitioning from honors to pass/fail grading with enhanced formative feedback in the core clerkship year may affect student learning experiences, wellbeing, and perceptions of the learning environment. Intervention: Core clerkship grading was transitioned from honors/pass/fail to pass/fail at one US medical school. In addition, the requirement for students to obtain formative supervisor feedback was formalized to twice per week. Context: This qualitative study utilized semi-structured interviews to explore the perceptions among core clerkship students of learning and assessment. Interview questions addressed motivation, wellbeing, learning behaviors, team dynamics, feedback, and student and supervisor attitudes regarding assessment changes. The authors analyzed data inductively using thematic analysis informed by sensitizing concepts related to theories of motivation (goal orientation theory and self-determination theory). Impact: Eighteen students participated, including five with experience in both honors-eligible and pass/fail clerkships. The authors identified three major themes in students' descriptions of the change in approach to assessment: student engagement in clerkships, wellbeing, and recognition of learning context. Student engagement subthemes included intrinsic motivation for patient care rather than performing; sense of agency over learning, including ability to set learning priorities, seek and receive feedback, take learning risks, and disagree with supervisors, and collaborative relationships with peers and team members. Positive wellbeing was characterized by low stress, sense of authenticity with team members, prioritized physical health, and attention to personal life. Learning context subthemes included recognition of variability of clerkship contexts with pass/fail grading mitigating fairness and equity concerns, support of the grading change from residents and some attendings, and implications surrounding future stress and residency selection. Lessons Learned: Students perceive a transition from honors grading to pass/fail with increased feedback as supporting their engagement in learning, intrinsic motivation, and wellbeing. Drivers of wellbeing appear to include students' feelings of control, achieved through the ability to seek learning opportunities, teaching, and constructive feedback without the perceived need to focus on impressing others. Ongoing evaluation of the consequences of this shift in assessment is needed.

Medical School to Residency: How Can We Trust the Process?

Background

To say that the transition from undergraduate medical education (UME) to graduate medical education (GME) is under scrutiny would be an understatement. Findings from a panel discussion at the 2018 Association of American Medical Colleges Annual meeting entitled, “Pass-Fail in Medical School and the Residency Application Process and Graduate Medical Education Transition” addressed what and when information should be shared with residency programs, and how and when that information should be shared.

Materials and Methods

Over 250 participants representing UME and GME (e.g. leadership, faculty, medical students) completed worksheets addressing these questions. During report-back times, verbal comments were transcribed in real time, and written comments on worksheets were later transcribed. All comments were anonymous. Thematic analysis was conducted manually by the research team to analyze the worksheet responses and report back comments.

Results

Themes based on suggestions of what information should be shared included the following: 1) developmental/assessment benchmarks such as demonstrating the ability/competencies to do clinical work; 2) performance on examinations; 3) grades and class ranking; 4) 360 evaluations; 5) narrative evaluations; 6) failures/remediation/gaps in training; 7) professionalism lapses; 8) characteristics of students such as resiliency/reliability; and 9) service/leadership/participation. In terms of how this information should be shared, the participants suggested enhancements to the current process of transmitting documents rather than alternative methods (e.g., video, telephonic, face-to-face discussions) and information sharing at both the time of the match and again near/at graduation to include information about post-match rotations.

Discussion

Considerations to address concerns with the transition from medical school to residency include further enhancements to the Medical Student Performance Evaluation, viewing departmental letters as ones of evaluation and not recommendation, a more meaningful educational handoff, and limits on the number of residency applications allowed for each student. The current medical education environment is ready for meaningful change in the UME to GME transition.

Beyond Medical Knowledge and Patient Care: A Program Director's Perspective for the Role of General Surgery Subinternships

OBJECTIVE

Established primary goals of general surgery subinternships are improvement in patient care and medical knowledge. However, the secondary gains such as obtaining recommendation letters and forming relationships are apparent but poorly defined. We sought the opinion of general surgery program directors (PDs) on the secondary purposes of subinternships. Our aim is to aide mentors and students to optimize the subinternship experience relative to the residency application process.

DESIGN

A survey was administered in July 2019. This questionnaire consisted of 11 items and was broken down into 3 sections: demographics, PD perspective on the secondary goals of a general surgery subinternship, and the role of general surgery subinternships on the students' residency application.

SETTING

An online, national survey through the Association for Program Directors listserv.

PARTICIPANTS

United States general surgery PDs affiliated with the Association for Program Directors listserv.

RESULTS

Sixty-one PDs completed the survey from 42 (69%) academic and 14 (23%) community programs. The majority of PDs (n = 33, 54%) reported that assessment of a subintern's suitability for their residency was the most important secondary purpose. Furthermore, PDs (n = 24, 39%) valued a letter of recommendation from faculty the student worked with during a subinternship as the most important criteria in the interview selection process. Away rotations were perceived as of equal value to subinternships completed at the student's home institution. Overall, PD opinions were similar at academic and community programs.

CONCLUSIONS

Our study suggests subinternships significantly impact a student's application to general surgery residency, clarifying a secondary role for these rotations. Subinternships are important for PDs to assess a student for ranking at their program. All students should pursue a letter of recommendation from subinternship faculty, when possible, as they can heavily influence the interview selection process. Away rotations should only be recommended for those students who need to strengthen their application.

Best Practices in Letters of Recommendation for General Surgery Residency: Results of Expert Stakeholder Focus Groups

OBJECTIVE

Performance inflation is rampant in applications to general surgery residency. The medical student performance evaluation, transcript, and letters of recommendation (LOR) have all been shown significantly biased in the applicants' favor. This study sought to determine best practices for LOR to improve transparency and alignment of applicant and program characteristics.

DESIGN

Two 1-hour focus groups were conducted using semi-structured interviews. Participants were asked to discuss the value and role of LOR characteristics, including standardized LOR, and provide recommendations for best practices. The transcribed discussions were coded by two educators using grounded theory and an inductive approach utilizing NVivo 12. Codes were then reviewed and revised to achieve consensus and recommendations.

SETTING

Focus groups were held during the annual Surgical Education Week meeting in April 2019.

PARTICIPANTS

General Surgery Program Directors from 10 institutions and Surgery Clerkship Directors from 11 other medical schools participated, with each group meeting independently from the other.

RESULTS

Individually, 18 codes were identified by the authors, with consensus agreed on ten. These were grouped into 4 themes: author factors, letter content, bias, and standardized letters. Overall, a checkbox and short-answer standardized LOR was not recommended, favoring a template of items to include and exclude. Ideal letter writers were felt to be surgeons who best know the applicant, and the Chair's letter, when they have no working knowledge of the applicant, was perceived to add little value. Use of specific examples to demonstrate applicant characteristics were favored, and descriptors for coded language should be included to aid in interpretation.

CONCLUSION

The focus groups identified best practices to guide writing LOR in support of applicants to general surgery residency. A template of content is provided to improve the efficiency, transparency, and accuracy of these letters for the benefit of students, medical schools, and residency programs.

Resident Selection in the Wake of United States Medical Licensing Examination Step 1 Transition to Pass/Fail Scoring

INTRODUCTION

The numeric score for the United States Medical Licensing Examination Step 1 is one of the only universal, objective, scaled criteria for comparing the many students who apply to orthopaedic surgery residency. However, on February 12, 2020, it was announced that Step 1 would be transitioning to pass/fail scoring. The purpose of this study was to (1) determine the most important factors used for interview and resident selection after this change and (2) to assess how these factors have changed compared with a previous report on resident selection.

METHODS

A survey was distributed to the program directors (PDs) of all 179 orthopaedic surgery programs accredited by the Accreditation Council for Graduate Medical Education. Questions focused on current resident selection practices and the impact of the Step 1 score transition on expected future practices.

RESULTS

A total of 78 PDs (44%) responded to the survey. Over half of PDs (59%) responded that United States Medical Licensing Examination Step 2 clinical knowledge (CK) score is the factor that will increase most in importance after Step 1 transitions to pass/fail, and 90% will encourage applicants to include their Step 2 CK score on their applications. The factors rated most important in resident selection from zero to 10 were subinternship performance (9.05), various aspects of interview performance (7.49 to 9.01), rank in medical school (7.95), letters of recommendation (7.90), and Step 2 CK score (7.27). Compared with a 2002 report, performance on manual skills testing, subinternship performance, published research, letters of recommendations, and telephone call on applicants' behalf showed notable increases in importance.

DISCUSSION

As Step 2 CK is expected to become more important in the residency application process, current applicant stress on Step 1 scores may simply move to Step 2 CK scores. Performance on subinternships will remain a critical aspect of residency application, as it was viewed as the most important resident selection factor and has grown in importance compared with a previous report.