The association between United States Medical Licensing Examination scores and clinical performance in medical students

Race and Gender Bias in Clerkship Grading

Phenomenon: Existing literature, as well as anecdotal evidence, suggests that tiered clinical grading systems may display systematic demographic biases. This study aimed to investigate these potential inequities in-depth. Specifically, this study attempted to address the following gaps in the literature: (1) studying grades actually assigned to students (as opposed to self-reported ones), (2) using longitudinal data over an 8-year period, providing stability of data, (3) analyzing three important, potentially confounding covariates, (4) using a comprehensive multivariate statistical design, and (5) investigating not just the main effects of gender and race, but also their potential interaction. Approach: Participants included 1,905 graduates (985 women, 51.7%) who received the Doctor of Medicine degree between 2014 and 2021. Most of the participants were white (n = 1,310, 68.8%) and about one-fifth were nonwhite (n = 397, 20.8%). There were no reported race data for 10.4% (n = 198). To explore potential differential grading, a two-way multivariate analysis of covariance was employed to examine the impact of race and gender on grades in eight required clerkships, adjusting for prior academic performance. Findings: There were two significant main effects, race and gender, but no interaction effect between gender and race. Women received higher grades on average on all eight clerkships, and white students received higher grades on average on four of the eight clerkships (Medicine, Pediatrics, Surgery, Obstetrics/Gynecology). These relationships held even when accounting for prior performance covariates. Insights: These findings provide additional evidence that tiered grading systems may be subject to systematic demographic biases. It is difficult to tease apart the contributions of various factors to the observed differences in gender and race on clerkship grades, and the interactions that produce these biases may be quite complex. The simplest solution to cut through the tangled web of grading biases may be to move away from a tiered grading system altogether.

Residency Program Directors' Views on Research Conducted During Medical School: A National Survey

PURPOSE

With the United States Medical Licensing Examination Step 1 transition to pass/fail in 2022, uncertainty exists regarding how other residency application components, including research conducted during medical school, will inform interview and ranking decisions. The authors explore program director (PD) views on medical student research, the importance of disseminating that work, and the translatable skill set of research participation.

METHOD

Surveys were distributed to all U.S. residency PDs and remained open from August to November 2021 to query the importance of research participation in assessing applicants, whether certain types of research were more valued, productivity measures that reflect meaningful research participation, and traits for which research serves as a proxy. The survey also queried whether research would be more important without a numeric Step 1 score and the importance of research vs other application components.

RESULTS

A total of 885 responses from 393 institutions were received. Ten PDs indicated that research is not considered when reviewing applicants, leaving 875 responses for analysis. Among 873 PDs (2 nonrespondents), 358 (41.0%) replied that meaningful research participation will be more important in offering interviews. A total of 164 of 304 most competitive specialties (53.9%) reported increased research importance compared with 99 of 282 competitive (35.1%) and 95 of 287 least competitive (33.1%) specialties. PDs reported that meaningful research participation demonstrated intellectual curiosity (545 [62.3%]), critical and analytical thinking skills (482 [55.1%]), and self-directed learning skills (455 [52.0%]). PDs from the most competitive specialties were significantly more likely to indicate that they value basic science research vs PDs from the least competitive specialties.

CONCLUSIONS

This study demonstrates how PDs value research in their review of applicants, what they perceive research represents in an applicant, and how these views are shifting as the Step 1 exam transitions to pass/fail.

Job Performance of Medical Graduates With Compulsory Services in Underserved Rural Areas in China: A Cohort Study

BACKGROUND

China started a national program in 2010 to train qualified general practitioners with compulsory services program (CSP) in rural and remote areas. While the program has shown positive effects on staffing primary healthcare (PHC) in rural areas, very little is known about how well they perform. This study aims to evaluate the job performance of medical graduates from this program and the influence of program design on job performance.

METHODS

A cohort study was conducted with graduates from CSP and non-CSP (NCSP) from four medical universities in central and western China. Baseline and three waves of follow-up surveys were conducted from 2015-2020. The pass rate of China National Medical Licensing Examinations (NMLE) and self-reported job performance were used as measurements. Multivariable regressions were used to identify factors affecting job performance.

RESULTS

2154 medical graduates were included, with 1586 CSP and 568 NCSP graduates. CSP (90.6%) and NCSP (87.5%) graduates showed no difference in passing the NMLE (P=.153). CSP graduates reported similar job performance with NCSP graduates (CSP, 63.7; NCSP, 64.2); in the multivariable regression, CSP graduates scored 0.32 and 1.36 points lower in the total sample and graduates of 2015-2017, respectively, but not significantly. Having formally funded positions improved the job performance of CSP (β coefficient=4.87, P<.05). After controlling for Qinghai which adopted a different contracting strategy, "working in hometown" showed significant influence on job performance (β coefficient = 1.48, P<.05).

CONCLUSION

CSP graduates have demonstrated as good job performance as NCSP, proving the competency to provide high-quality care for remote and rural areas. The contracted township health centers (THCs) should provide guidance for CSP graduates, especially in the first few years after graduation. The local government should provide formally funded positions on time and prioritize signing contracts with hometowns or places nearby.

Gender differences in repeat-year experience, clinical clerkship performance, and related examinations in Japanese medical students

While the number of female medical students is increasing in Japan, gender differences in medical school performance have not been studied extensively. This study aimed to compare gender differences in repeat-year experience, Clinical Clerkship (CC) performance, and related examinations in Japanese medical students. We retrospectively analyzed the number of repeat-year students and years to graduation for male and female medical students, and assessed gender differences in performance on computer-based testing (CBT) before CC, CC as evaluated by clinical teachers, the CC integrative test, and the graduation examination in 2018-2020 graduates from our medical school. Subgroup analyses excluding repeat-year students were also performed. From 2018 to 2020, 328 medical students graduated from our medical school. There were significantly fewer repeat-year female students compared to male students (P = .010), and the average number of years to graduate was significantly higher for male students than female students (P < .001). Female students showed higher scores and performance in all integrative tests and CC (P < .05, each). In analysis excluding repeat-year students, there were no significant gender difference in performance on the CBT, and CC integrative test, although female students significantly outperformed male students on the CC and graduation examination. Female medical students had a fewer number of repeat-years and performed better in the CC and graduation examination compared to their male counterparts.

Current Views on the New United States Medical Licensing Examination Step 1 Pass/Fail Format: A Review of the Literature

INTRODUCTION

Residency programs have historically used numerical Step 1 scores to screen applicants, making it a career-defining, high-stakes examination. Step 1 scores will be reported as pass/fail starting in January 2022, fundamentally reshaping the residency application review process. This review aimed to identify opinions of physicians and medical students about the new format, identify arguments in support of or against the change, and determine the implications of this change on the residency selection process.

METHODS

A comprehensive PubMed review was performed in May 2021 to identify articles that discussed the new Step 1 format. Non-English and duplicate articles were excluded. Data collected from each article included publication year, specialty, subjects, and key findings.

RESULTS

A total of 81 articles were included, 26 of which discussed the impact of the new format within surgical fields (32.1%). Remaining articles discussed the implications within the medical community as a whole (n = 33, 40.7%) and nonsurgical fields (n = 22, 27.2%). Studies suggest Program Directors will rely on Step 2 Clinical Knowledge (CK) scores, medical school reputation, applicant familiarity, Dean's letters, recommendation letters, and research in lieu of numerical Step 1 scores. In addition, concerns have been raised that the new format will disadvantage international, osteopathic, and minority applicants while increasing stress surrounding Step 2 CK.

CONCLUSIONS

Within the medical community, there are concerns that Step 2 CK will be used to substitute Step 1 and that resident diversity will diminish due to the new Step 1 format. Holistic candidate consideration will be increasingly important.

Characteristics of First-Year Residents in Top-Ranked United States Ophthalmology Residency Programs

Objective The aim of the study is to investigate the characteristics of first-year residents associated with attending a top-ranked United States (U.S.) ophthalmology residency program over the past decade.

Methods First-year ophthalmology residents in 2009, 2013, 2016, and 2019 were identified from institutional websites, Doximity, LinkedIn and the Wayback Machine. Publications were obtained from Scopus and Google Scholar; research productivity was measured using the h-index, and medical school region based on U.S. Census Bureau designations. Medical school and ophthalmology training program rankings were based on U.S. News & World Report (U.S. News) rankings and National Institutes of Health (NIH) funding. One-way ANOVA, Wilcoxon rank sum, χ 2, and t-tests were used to analyze trends, and odds ratios (ORs) were calculated using logistic regression.

Results Data were obtained on 81% (1,496/1,850) of the residents; 43% were female; 5% were international medical graduates (IMGs); and 10% had other graduate degrees. Over the decade, the mean h-index increased (0.87–1.26; p <0.05) and the proportion of residents who attended a top 20 medical school decreased (28–18%; p <0.05). In a multivariate logistic regression model, characteristics associated with being a first-year resident in a top 20 program ranked by U.S. News were female gender [OR: 1.32, 95% CI: 1.02–1.72], having a Master's degree [OR: 2.28, 95% CI: 1.29–4.01] or PhD [OR: 2.23, 95% CI: 1.32–3.79], attending a top 20 [OR: 5.26, 95% CI: 3.66–7.55] or a top 40 medical school by NIH funding [OR: 2.45, 95% CI: 1.70–3.54], attending a medical school with a mean USMLE Step 2 score above 243 [OR: 1.64, 95% CI: 1.01–2.67] or located in the Northeast [OR: 2.00, 95% CI: 1.38–2.89] and having an h-index of one or more [OR: 1.92, 95% CI: 1.47–2.51]. Except for gender, these characteristics were also significantly associated with matching to a top 20 ophthalmology program by NIH funding.

Conclusion Female gender, graduate degrees, research productivity, and attending a medical school with high research productivity, high mean USMLE Step 2 score or in the Northeast were key characteristics of first-year residents in top-ranked U.S. ophthalmology residency programs.

Enhanced Learning and Retention of Medical Knowledge Using the Mobile Flash card Application Anki

INTRODUCTION

As medical schools condense the basic science phase of undergraduate medical education, it has become increasingly important to identify methods and tools that facilitate learning, mastery, and application of medical knowledge. One increasingly popular tool that promotes engagement with content is Anki, a web-based flash card system. Using Anki, medical students can access pre-made flash cards specifically tailored to prepare students for the United States Medical Licensing Exam (USMLE) Step 1 exam. The objective of this study was to identify Anki use and its association to USMLE Step 1 performance.

METHODS

In March 2020, medical students in years 2, 3, and 4 who had completed USMLE Step 1 were administered a survey to measure Anki usage. The survey was locally developed and was reviewed by survey experts on campus. Survey responses were paired with USMLE Step 1 results for analyses. Descriptive and inferential statistics were used for analysis.

RESULTS

Anki usage was associated with higher USMLE Step 1 scores. Additionally, amongst those who used Anki, those with more consistent use had higher USMLE Step 1 scores and higher perceived levels of knowledge retention.

CONCLUSIONS

This research suggests that Anki is an effective educational tool that should be recommended to medical students alongside other evidenced-based study tools, such as the popular question bank USMLE World. Future research should attempt to identify a relationship between Anki usage and future clinical performance to demonstrate the implications that Anki has on clinical skills.

SUPPLEMENTARY INFORMATION

The online version contains supplementary material available at 10.1007/s40670-021-01386-9.

Inclusion and equity: Experiences of underrepresented in medicine physicians throughout the medical education continuum

The mission of pediatric medical education is to train the workforce who will provide and innovate child healthcare, including addressing healthcare inequities for children and adolescents of color. A diverse physician workforce is associated with more equitable outcomes. Therefore, it is imperative that our workforce continue to reflect the ever-growing diverse nature of our society. While diversity has been at the forefront of conversations for increasing the number of underrepresented communities in medicine, inclusion and equity are oftentimes overlooked. It is crucial to recognize that while diversity is an important aspect to the UIM experience, the overall impact of diversity is diluted when it is not paired with inclusion and equity. Therefore, the purpose of this paper is to illustrate the experiences that shape inclusion and equity for UIM physicians along the training continuum and how negative factors may impact the wellness and career longevity of UIM physicians. Our study explored the experiences of UIM physicians throughout their medical education through the lens of inclusion and equity by exploring three common factors: social support, racism and discrimination, and well-being and burnout. There may be a perception that stressors, including racial discrimination, diminish with progression through one's educational career; however, this review suggests that these stressors manifest differently relative to the trials inherent to each career stage. By exploring the interplay between these factors and the UIM experience as a continuum, we are better able to discuss effective solutions to diversity, equity and inclusion for UIM physicians.

Predicting United States Medical Licensing Examination Step 2 Clinical Knowledge Scores from Previous Academic Performance Measures within a Longitudinal Interleaved Curriculum

Background

United States Medical Licensing Examination (USMLE) Step 1 is a common metric looked at by residency programs to determine invitations for candidates to interview. However, USMLE Step 2 Clinical Knowledge (CK) has also been an important factor for selecting applicants to interview and plays a significant role during applicant selection. This study aims to identify academic performance measures that correlate with USMLE Step 2 CK scores and to develop a model to predict USMLE Step 2 CK scores using previous academic measures from the first two cohorts in the longitudinal interleaved clerkship (LInC) at the Kirk Kerkorian School of Medicine at the University of Nevada, Las Vegas (KSOM).

Setting

The KSOM is a newly accredited US allopathic medical school that accepted its first class in 2017. At KSOM, a LInC model is used in the primary clinical year. In this model, rotations are two weeks in duration before moving on to the next specialty. Students complete the National Board of Medical Examiners (NBME) subject examinations in all six specialties in one week at the midpoint and the end of the LInC. Students who passed an exam at the midpoint can opt out of that exam at the end as the higher of the two exam scores is recorded. However, most students choose to take all the exams again to improve their scores and prepare for USMLE Step 2 CK.

Methodology

Academic performance measures were gathered from the class of 2021 and 2022 (n = 101) including undergraduate grade point average (GPA), undergraduate science GPA, medical college admission test score, USMLE Step 1 score, NBME clinical subject exam scores, and USMLE Step 2 CK scores. Pearson correlations were run between the performance variables and USMLE Step 2 CK scores to measure influence variables individually, then a regression model measured impacts of variables together.

Results

All variables except undergraduate science GPA significantly correlated with USMLE Step 2 CK score. USMLE Step 1 had the strongest correlation (r = 0.752, p < 0.001). The regression model had an R of 0.859 with the internal medicine subject exam showing the highest beta coefficient (0.327, p < 0.001).

Conclusions

This study determined that USMLE Step 2 CK scores can be effectively predicted using available performance measures. With USMLE Step 1 becoming pass/fail in January 2022, the importance of USMLE Step 2 CK as a screening tool in the residency application process will likely increase. This study was conducted within a LInC curriculum and may have limited value in the prediction of scores within other clinical year curricula.

Medical Students' Reflections on the Recent Changes to the USMLE Step Exams

The United States Medical Licensing Examination (USMLE) consists of Step 1, Step 2 Clinical Knowledge, Step 2 Clinical Skills, and Step 3. To be licensed to practice medicine in the United States, medical students must pass all parts of the USMLE. However, in addition to that pass/fail grade, students are currently given a numerical score for Step 1, Step 2 Clinical Knowledge, and Step 3. Residency program directors have come to use the Step 1 score to efficiently screen a growing number of residency applicants. As a result, a deleterious environment in undergraduate medical education has been created, given the importance of Step 1 to medical students matching to their preferred residency program. It was announced in February 2020 that the score-reporting protocol for Step 1 would be changed from a 3-digit numerical score to pass/fail only, beginning no earlier than January 1, 2022. This decision will undoubtedly impact medical students, medical schools, and residency program directors. Here, the authors discuss the impact that the change to Step 1 scoring will have on these key stakeholder groups, from their perspective as students at MD-granting medical schools in the United States. They also call attention to outstanding issues with the USMLE that must be addressed to improve undergraduate medical education for all stakeholders, and they offer advice for further improvements to the residency application process.

Correlation of Medical College Admission Test Scores and Self-assessment Materials with the United States Medical Licensing Examination Step 1 Performance

Purpose

Candidates' performance on the United States Medical Licensing Examination (USMLE) Step 1 examination had been correlated with the Medical College Admission Test (MCAT). However, in 2015, a new MCAT format was released and its correlation with Step 1 remains to be fully analyzed. Preparation for Step 1 typically involves purchasing and perusing practice tests from the National Board of Medical Examiners (NBME) and UWorld; however, their predictive value to performance on Step 1 remains to be ascertained, especially with the release of five new NBME practice tests. Additionally, there is a need for accurately predicting Step 1 scores to self-evaluate study progress and reduce student anxiety.

Rationale

Program directors rank USMLE Step 1 scores as the number one criterion in selecting interviewees for residency. Step 1 scores are more important than Step 2 scores, Dean’s letter, or other letters of recommendation in determining the overall ranking of a candidate after interviews.

Hypotheses

The authors hypothesized that the new MCAT scores correlated positively with Step 1 scores and that the new NBME practice tests were more predictive of performance on Step 1 as compared to old NBME tests. 

Methods

Linear regression analysis followed by either analysis of variance (ANOVA) or Student's t-tests were used to analyze 399 responses. Data obtained was used to update an existing Step 1 score predictor, which was then validated.

Results

A positive correlation between the MCAT (average score: 510.1 ± 6.3) and Step 1 scores (average score: 246.1 ± 14.2) was observed. While new NBME practice tests were more predictive of Step 1 scores than old NBME tests, UWorld test scores were the most predictive. Students who practiced with the new NBME practice tests scored significantly higher than students who did not use them. However, students using any of the UWorld practice tests did significantly better than students who practiced using only NBME practice tests but not UWorld practice tests. Ironically, NBME16,the second-most correlativetest to Step 1 performance, is no longer available for purchase. Overall, taking six or more practice tests significantly enhanced Step 1 scores; the optimal number of tests was found to be between six and nine. The predicted score by an updated Step 1 score predictor was within 3.8 points or 1.6% of the actual Step 1 score.

Conclusions

We believe this study will aid in the selection and purchase of appropriate self-assessment tests as preparatory material for the USMLE Step 1 examination. It will also introduce them to an existing Step 1 score predictor that will help determine their readiness for Step 1.