Improving Residency Selection Requires Close Study and Better Understanding of Stakeholder Needs

Resident Recruitment in a New Era

ABSTRACT

This chapter focuses on resident recruitment and recent US National Resident Matching Program changes and the impact in the evaluation and ranking of applicants within the specialty of anesthesiology. Recruitment challenges are examined as well as program strategies and potential future directions. Also discussed are DEI initiatives within the recruitment process.

The USMLE® STEP 1 Pass or Fail Era of the Vascular Surgery Residency Application Process: Implications for Structural Bias and Recommendations

MOTIVES BEHIND THE SHIFT

USMLE® STEP 1 score reporting has been changed to a binary pass/fail format since January 26, 2022. The motives behind this change were (1) the questionable validity of using USMLE® STEP 1 as a screening tool during the candidate selection process and (2) the negative impact of using standardized examination scores as an initial gatekeeping threshold for the underrepresented in medicine (URiM) candidates applying to GME programs, given their generally lower mean standardized exams scores compared to non-URiM students. The USMLE® administrators justified this change as a tactic to enhance the overall educational experience for all students and to increase the representation of URiM groups. Moreover, they advised the program directors (PDs) to give more attention to other important qualities and components such as the applicant's personality traits, leadership roles and other extracurricular accomplishments, as part of a holistic evaluation strategy. IMPACT OF USMLE® STEP 1 PASS/FAIL: At this early stage, it is unclear how this change will impact Vascular Surgery Integrated residency (VSIR) programs. Several questions are outstanding, most importantly, how VSIR PDs will evaluate applicants absent the variable which heretofore was the primary screening tool. Our previously published survey showed that VSIR PDs will move their attention to other measures such as USMLE® STEP 2CK and letters of recommendation during the VSIR selection process. Furthermore, more emphasis on subjective measures such as the applicant's medical school rank and extracurricular student activities is expected. Given the expected higher weight of USMLE® STEP 2CK in the selection process than ever, many anticipate that medical students will dedicate more of their limited time to its preparation at the expense of both clinical and non-clinical activities. Potentially leaving less time to explore specialty pathways and to determine whether VS is the appropriate career for them.

A FRAMEWORK FOR CANDIDATE EVALUATION

The critical juncture in the VSIR candidate evaluation paradigm presents an opportunity to thoughtfully transform the process via current (Standardized Letter of Recommendation, USMLE® STEP 2CK, and clinical research) and future (Emotional Intelligence, Structure Interview and Personality Assessment) measures which constitute a framework to follow in the USMLE® STEP 1 pass/fail era.

Quality assurance in health professions education: Role of accreditation and licensure

OBJECTIVE

The aim of this paper is to provide an overview of the major quality assurance strategies, accreditation and licensure, in health professions education. It explores the nature of these regulatory processes using Brazil and the United States as examples because these large systems are at different ends of the developmental continuum. For each, it describes the tensions that arise, offers a critical synthesis of the evidence and maps out future directions.

RESULTS

Given wide variability among operating medical schools in curricular design, length of study, resources and facilities for clinical training and supervision, the nature of regulatory bodies varies considerably. Nonetheless, they share tensions related purpose and process including quality assurance versus quality improvement, outcomes versus process and continuous versus episodic evaluations and assessments. Clear evidence of effectiveness, especially for accreditation, is scarce and difficult to obtain, particularly as it relates to health outcomes.

CONCLUSIONS

Regulatory processes need to be built around clear definitions of the goals for each stage of professional development, the current movement towards competency-based education and the variable durations of medical education. These changes must motivate revisions in the content and process of programmes for accreditation and licensure, complimentary efforts towards quality of care, and stimulate a significant research effort.

Optimizing the residency application process: insights from neurological surgery during the pandemic virtual application cycle

OBJECTIVE

In this article, the authors describe the impact of the COVID-19 virtual match cycle and discuss approaches to optimize future cycles through applicant and neurosurgical education leadership insights.

METHODS

Anonymous surveys of neurosurgery program leaders (program directors and program chairs), program administrators (PAs), and 2020-2021 neurosurgery residency match applicants were distributed by the SNS, in conjunction with the Association of Resident Administrators in Neurological Surgery and AANS Young Neurosurgeons Committee.

RESULTS

Responses were received from 77 (67.0%) of 115 PAs, 119 (51.7%) of 230 program leaders, and 124 (44.3%) of 280 applicants representing geographically diverse regions. During the virtual application cycle relative to the previous year, programs received more Electronic Residency Application Service applications (mean 314.8 vs 285.3, p < 0.0001) and conducted more applicant interviews (mean 45.2 vs 39.9, p = 0.0003). More than 50% of applicants applied to > 80 programs; 60.3% received ≤ 20 interview invitations, and 9% received > 40 invitations. Overall, 65% of applicants completed ≤ 20 interviews, whereas 34.7% completed > 20 interviews. Program leaders described one 4-week home subinternship (93.3%) and two 4-week external subinternships (68.9%) as optimal neurosurgical exposure; 62.8% of program leaders found the standardized letter of recommendation template to be somewhat (47.5%) or significantly (15.3%) helpful. Applicants, PAs, and program leaders all strongly preferred a hybrid model of in-person and virtual interview options for future application cycles over all in-person or all virtual options. Ninety-three percent of applicants reported matching within their top 10-ranked programs, and 52.9% of programs matched residents within the same decile ranking as in previous years.

CONCLUSIONS

Optimizing a national strategy for the neurosurgery application process that prioritizes equity and reduces costs, while ensuring adequate exposure for applicants to gain educational opportunities and evaluate programs, is critical to maintain a successful training system.

Machine Learning for The Prediction of Ranked Applicants and Matriculants to an Internal Medicine Residency Program

Phenomenon: Residency programs throughout the country each receive hundreds to thousands of applications every year. Holistic review of this many applications is challenging, and to-date, few tools exist to streamline or assist in the process for selecting candidates to interview and rank. Machine learning could assist programs in predicting which applicants are likely to be ranked, and among ranked applicants, which are likely to matriculate.Approach: In the present study, we used the machine learning algorithm Random Forest (RF) to differentiate between ranked and unranked applicants as well as matriculants and ranked non-matriculants to an internal medicine residency program in northern New England over a three-year period. In total, 5,067 ERAS applications were received during the 2016-17, 2017-18, and 2018-19 application cycles. Of these, 4,256 (84.0%) were unranked applicants, 754 (14.9%) were ranked non-matriculants, and 57 (1.12%) were ranked matriculants.Findings: For differentiating between ranked and unranked applicants, the RF algorithm achieved an area under the receiver operating characteristic (AUROC) curve of 0.925 (95% CI: 0.918-0.932) and area under the precision-recall curve (AUPRC) of 0.652 (0.611-0.685), while for differentiating between matriculants and ranked non-matriculants, the AUROC was 0.597 (95% CI: 0.516-0.680) and AUPRC was 0.114 (0.075-0.167). The ranks of matriculated applicants were significantly higher using the algorithmic rank list as compared with the actual rank list for the 2017-18 (median rank: 98 versus 204, p < .001) and 2018-19 cycles (74 versus 192, p = .006), but not the 2016-17 cycle (97 versus 144, p = .37).Insights: The RF algorithm predicted which applicants among the overall applicant pool were ranked with impressive accuracy and identified matriculants among ranked candidates with modest but better-than-random accuracy. This approach could assist residency programs with triaging applicants based on the likelihood of a candidate being ranked and/or matriculating.

Current Views on the New United States Medical Licensing Examination Step 1 Pass/Fail Format: A Review of the Literature

INTRODUCTION

Residency programs have historically used numerical Step 1 scores to screen applicants, making it a career-defining, high-stakes examination. Step 1 scores will be reported as pass/fail starting in January 2022, fundamentally reshaping the residency application review process. This review aimed to identify opinions of physicians and medical students about the new format, identify arguments in support of or against the change, and determine the implications of this change on the residency selection process.

METHODS

A comprehensive PubMed review was performed in May 2021 to identify articles that discussed the new Step 1 format. Non-English and duplicate articles were excluded. Data collected from each article included publication year, specialty, subjects, and key findings.

RESULTS

A total of 81 articles were included, 26 of which discussed the impact of the new format within surgical fields (32.1%). Remaining articles discussed the implications within the medical community as a whole (n = 33, 40.7%) and nonsurgical fields (n = 22, 27.2%). Studies suggest Program Directors will rely on Step 2 Clinical Knowledge (CK) scores, medical school reputation, applicant familiarity, Dean's letters, recommendation letters, and research in lieu of numerical Step 1 scores. In addition, concerns have been raised that the new format will disadvantage international, osteopathic, and minority applicants while increasing stress surrounding Step 2 CK.

CONCLUSIONS

Within the medical community, there are concerns that Step 2 CK will be used to substitute Step 1 and that resident diversity will diminish due to the new Step 1 format. Holistic candidate consideration will be increasingly important.

Innovation in Resident Selection: Life Without Step 1

INTRODUCTION

The announcement of Step 1 shifting to a Pass/Fail metric has prompted resident selection committees (RSCs) to pursue objective methods of evaluating prospective residents. Regardless of the program's specialty or affiliated hospital/school, RSCs universally aim to recognize and choose applicants who are an "optimal fit" to their programs.¹ An optimal fit can be defined as a candidate who thrives in the clinical and academic setting, both contributing to and benefiting from their respective training environments.

OBJECTIVE

The objective of this scoping review is to evaluate alternative, innovative methods by which RSCs can evaluate applicants and predict success during residency. Objective methods include: Step 2 scores, Traditionally Used Metrics (core clerkship scores), interview performance, musical talent, sports involvement, AOA membership, research publications, unprofessional behavior, Dean's letters, Rank list, judgement testing, and specialty-specific shelf exams.^13-15.

METHODS

A scoping review was performed in compliance with the guidelines indicated by the PRISMA Protocol for scoping review.¹⁸ 9308 results were identified in the original PubMed search for articles with the key words "Resident Success". Abstract screening and application of inclusion and exclusion criteria yielded 97 articles that were critically appraised via review of full manuscript.

RESULTS

Of the articles that focused on personality traits, situational judgement testing, and specialty specific pre-assessment, all of them demonstrated some level of predictability for resident success. Standardized Letter of Recommendations, Traditionally Used Metrics, and STEP 2 did not show a unanimous consensus in demonstrating predictability of a resident's success, this is because some articles suggested predictability and some articles disputed predictability.

CONCLUSION

The authors found personality traits, situational judgement testing, and specialty specific assessments to be predictive in selecting successful residents. Further research should aim to analyze exactly how RSCs utilize these assessment tools to aid in screening their large and competitive applicant pools to find residents that will be successful in their program.

Racial disparities in vascular surgery: An analysis of race and ethnicity among U.S. medical students, general surgery residents, vascular surgery trainees, and the vascular surgery workforce

The ever-increasing diversity of the U.S. population has resulted in a challenge to a vascular surgery workforce that does not represent the national demographics. Educational institutions, medical organizations, and medical agencies, through initiatives and programs, have attempted to increase the number of underrepresented minorities, with, however, only modest changes during the past 30 years. Several obstacles have been identified, some of which include racism, issues of finances, lack of mentors and scholarly activities, and inequity in promotion. In the present study, we have reviewed the reported data addressing these concerns and provided guidance toward the amelioration of these issues with the hope of constructive change toward encouraging a diverse vascular surgery workforce.

Stakeholder Perspectives on Standardizing the Residency Application and Interview Processes

OBJECTIVE

The purpose of this study was to examine stakeholder perspectives on recommended standards for the obstetrics and gynecology (OBGYN) residency application and interview processes proposed for the 2019 to 2020 application cycle. The authors aimed to assess the acceptance and perception of key stakeholders on the feasibility of implementing the standards as well as the effect of these changes on applicant anxiety.

DESIGN AND SETTING

The authors electronically distributed an anonymous survey in February 2020 to OBGYN residency applicants, clerkship directors, student affairs deans, program directors, and program managers. Participants received a 15-item survey, with questions assessing the importance and adoption of the guidelines, as well as their effect on perceived applicants' anxiety. Responses were measured on a 5-item Likert scale. Multiple regression analysis was used to explore which residency factors were associated with compliance with the standards. IRB exemption was granted by the University of Michigan.

PARTICIPANTS

A total of 1358 participants completed the survey for an overall response rate of 39.26%. Response rates were 36.04% for applicants (904/2508), 46.67% for CDs (105/225), 34.84% for members of GSAs (34/155), 59.43% for program directors (167/281), and 51.03% for program managers (148/290).

RESULTS

The overall response rate was 39.26% (1358/3459) with 36.04% of applicants (904/2508), 46.67% of clerkship directors (105/225), 34.84% of student affairs deans (34/155), 59.43% of program directors (167/281), and 51.03% of program managers (148/290). The recommendations were perceived as important by all stakeholders. More than 90% of program directors reported compliance with some or all of the recommendations and more than 90% of all applicants, clerkship directors and student affairs deans reported that the standards reduced applicant anxiety. All stakeholders rated each guideline to be important to extremely important.

CONCLUSIONS

This study demonstrates the feasibility and acceptance of universal standards for the residency application process in the field of OBGYN. The vast majority of stakeholders surveyed supported the initiative and participated in the guidelines. Applicant respondents perceived the guidelines to be important and to decrease anxiety surrounding the application and interview timelines. These findings are important for other specialties when considering similar interventions.

Systems-Level Reforms to the US Resident Selection Process: A Scoping Review

BACKGROUND

Calls to reform the US resident selection process are growing, given increasing competition and inefficiencies of the current system. Though numerous reforms have been proposed, they have not been comprehensively cataloged.

OBJECTIVE

This scoping review was conducted to characterize and categorize literature proposing systems-level reforms to the resident selection process.

METHODS

Following Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) guidelines, searches of Embase, MEDLINE, Scopus, and Web of Science databases were performed for references published from January 2005 to February 2020. Articles were included if they proposed reforms that were applicable or generalizable to all applicants, medical schools, or residency programs. An inductive approach to qualitative content analysis was used to generate codes and higher-order categories.

RESULTS

Of 10 407 unique references screened, 116 met our inclusion criteria. Qualitative analysis generated 34 codes that were grouped into 14 categories according to the broad stages of resident selection: application submission, application review, interviews, and the Match. The most commonly proposed reforms were implementation of an application cap (n = 28), creation of a standardized program database (n = 21), utilization of standardized letters of evaluation (n = 20), and pre-interview screening (n = 13).

CONCLUSIONS

This scoping review collated and categorized proposed reforms to the resident selection process, developing a common language and framework to facilitate national conversations and change.

USMLE Step 1 Is Pass/Fail - Should the ABSITE Follow Suit?

The United States Medical Licensing Examination (USMLE) Step 1 was recently made pass/fail. This decision was controversial largely because of the reliance on USMLE Step 1 scores in resident selection. However, these scores do not correlate with resident ability. In this manuscript, we consider if the American Board of Surgery In-Training Examination (ABSITE) should be pass/fail as well. The ABSITE has been used for "high-stakes" purposes, such as preliminary resident advancement and prospective fellow evaluation, for which it was not intended. Moreover, similar to the USMLE Step 1 exam, ABSITE scores have demonstrated no correlation with clinical ability. A pass/fail ABSITE would return the exam to its original purpose and minimize an over-reliance on scores. Moving forward, new objective measures will need to be developed to assess surgical trainees in a more holistic manner.

The Case for Capping Residency Interviews

OBJECTIVE

To determine 2020 residency cycle application practices and to model potential consequences in the 2021 cycle if (1) applicants scheduled an uncapped number of interviews; (2) applicants were capped to schedule 12 interviews; (3) residency programs kept their number of interview offers unchanged; and (4) programs increased their interview offers by 20%.

DESIGN AND SETTING

The authors sent an anonymous survey to all obstetrics and gynecology applicants registered through the Electronic Residency Application Service in February 2020 asking respondents to share demographics and number of interview offers received and completed. Based on prior estimates that 12 interviews are needed to match in obstetrics and gynecology, respondents were divided into Group 12+ (those receiving ≥12 interview offers) and Group <12 (those receiving <12 offers). Model assumptions were (1) applicants can complete all interviews they are offered because they are virtual; (2) interview offers that applicants in Group 12+ decline are subsequently offered to applicants in Group <12; (3) the proportions of interviews offered to Group 12+ and Group <12 will remain the same if programs chose to increase their total number of interview spots.

PARTICIPANTS

Among 2508 applicants, 750 (30%) provided the number of interview offers received and completed: 417 (56%) in Group 12+ and 333 (44%) in Group <12.

RESULTS

In models where applicants are uncapped in the number of interviews, Group <12 applicants receive <1 interview offer, even if programs increase the number of interviews offered and performed. If applicants are capped at 12 interviews, Group <12 applicants will receive 9 interview offers on average and will reach 12 if programs increase the number of interviews offered by 20%.

CONCLUSIONS

This work highlights how current inefficiencies may lead to negative consequences with virtual interviews. Interview caps and preference signaling systems need to be urgently considered.

A Model for Exploring Compatibility Between Applicants and Residency Programs: Right Resident, Right Program

Holistic review of residency applications is touted as the gold standard for selection, yet vast application numbers leave programs reliant on screening using filters such as United States Medical Licensing Examination scores that do not reliably predict resident performance and may threaten diversity. Applicants struggle to identify which programs to apply to, and devote attention to these processes throughout most of the fourth year, distracting from their clinical education. In this perspective, educators across the undergraduate and graduate medical education continuum propose new models for student-program compatibility based on design thinking sessions with stakeholders in obstetrics and gynecology education from a broad range of training environments. First, we describe a framework for applicant-program compatibility based on applicant priorities and program offerings, including clinical training, academic training, practice setting, residency culture, personal life, and professional goals. Second, a conceptual model for applicant screening based on metrics, experiences, attributes, and alignment with program priorities is presented that might facilitate holistic review. We call for design and validation of novel metrics, such as situational judgment tests for professionalism. Together, these steps could improve the transparency, efficiency and fidelity of the residency application process. The models presented can be adapted to the priorities and values of other specialties.

Medical Student Attitudes toward USMLE Step 1 and Health Systems Science - A Multi-Institutional Survey

Phenomenon: Because of its importance in residency selection, the United States Medical Licensing Examination Step 1 occupies a critical position in medical education, stimulating national debate about appropriate score use, equitable selection criteria, and the goals of undergraduate medical education. Yet, student perspectives on these issues and their implications for engagement with health systems science-related curricular content are relatively underexplored. Approach: We conducted an online survey of medical students at 19 American allopathic medical schools from March-July, 2019. Survey items were designed to elicit student opinions on the Step 1 examination and the impact of the examination on their engagement with new, non-test curricular content related to health systems science. Findings: A total of 2856 students participated in the survey, representing 23.5% of those invited. While 87% of students agreed that doing well on the Step 1 exam was their top priority, 56% disagreed that studying for Step 1 had a positive impact on engagement in the medical school curriculum. Eighty-two percent of students disagreed that Step 1 scores should be the top item residency programs use to offer interviews. When asked whether Step 1 results should be reported pass/fail with no numeric score, 55% of students agreed, while 33% disagreed. The majority of medical students agreed that health systems science topics were important but disagreed that studying for Step 1 helped learn this content. Students reported being more motivated to study a topic if it was on the exam, part of a course grade, prioritized by residency program directors, or if it would make them a better physician in the future. Insights: These results confirm the primacy of the United States Medical Licensing Examination Step 1 exam in preclinical medical education and demonstrate the need to balance the objectives of medical licensure and residency selection with the goals of the broader medical profession. The survey responses suggest several potential solutions to increase student engagement in health systems science curricula which may be especially important after Step 1 examination results are reported as pass/fail.

Residency match interview scheduling: quantifying the applicant experience

BACKGROUND

The process of offering and scheduling residency interviews varies widely among programmes. Applicants report distress and have advocated for reform. However, there is a paucity of quantitative data to characterise applicant concerns.

OBJECTIVE

We quantified the interview scheduling experience for US allopathic medical students in the 2020 main residency match.

METHODS

An anonymous, 13 question survey was sent to student representatives from each Association of American Medical Colleges member institution. Recipients were asked to forward the survey to their entire fourth-year class.

RESULTS

Of 4314 applicants to whom the survey was sent, 786 (18.2%) responded. Overall, 20.4% reported missing the opportunity to interview at a programme because they did not have adequate time to respond to an invitation; applicants into surgical specialties were significantly more likely than their non-surgical peers to report this experience (26.4% vs 18.4%, p<0.05). Most (57.4%) respondents scheduled an interview knowing they would likely cancel it in the future. The most commonly cited reason for this behaviour was concern that applicants would not receive invitations from other programmes (85.6%). A majority (56.4%) of respondents did not believe the match interview process functions based on equity and merit.

CONCLUSIONS

About one in five respondents missed the opportunity to interview at a programme because they did not respond to an invitation in time. Most respondents scheduled interviews knowing that they were likely to cancel them in the future. Standardisation of the interview invitation timeline would address these concerns.

Medical Students' Reflections on the Recent Changes to the USMLE Step Exams

The United States Medical Licensing Examination (USMLE) consists of Step 1, Step 2 Clinical Knowledge, Step 2 Clinical Skills, and Step 3. To be licensed to practice medicine in the United States, medical students must pass all parts of the USMLE. However, in addition to that pass/fail grade, students are currently given a numerical score for Step 1, Step 2 Clinical Knowledge, and Step 3. Residency program directors have come to use the Step 1 score to efficiently screen a growing number of residency applicants. As a result, a deleterious environment in undergraduate medical education has been created, given the importance of Step 1 to medical students matching to their preferred residency program. It was announced in February 2020 that the score-reporting protocol for Step 1 would be changed from a 3-digit numerical score to pass/fail only, beginning no earlier than January 1, 2022. This decision will undoubtedly impact medical students, medical schools, and residency program directors. Here, the authors discuss the impact that the change to Step 1 scoring will have on these key stakeholder groups, from their perspective as students at MD-granting medical schools in the United States. They also call attention to outstanding issues with the USMLE that must be addressed to improve undergraduate medical education for all stakeholders, and they offer advice for further improvements to the residency application process.

Perspectives on Transitioning Step 1 of the United States Medical Licensing Examination to a Pass/Fail Scoring Model: Defining New Frameworks for Medical Students Applying for Residency

For years, the USMLE Step 1 has acted as an unofficial "concours" for medical students applying to residency positions in the United States. The three-digit numeric score has been used to rank thousands of applicants without any evidence of validity. The USMLE will soon change score reporting to a pass/fail outcome. The main reason given was to address the concerns about its effects on the well-being of the students and medical education. It is argued that time for change has come. The authors discuss the various viewpoints of the stakeholders and the effects of this change on applicants and potential changes on the undergraduate medical curriculum. Furthermore, this article discusses several metrics that can be utilized in the application process in lieu of the USMLE Step 1. Additionally, some novel key metrics in the application process are identified, and their unique dynamic and adaptive characteristics are deliberated. Finally, the benefits of a transparent and holistic process are strongly advocated.

On Step 1 Mania, USMLE Score Reporting, and Financial Conflict of Interest at the National Board of Medical Examiners

Though intended to inform a binary decision on initial medical licensure, the United States Medical Licensing Examination (USMLE) is frequently used for screening candidates for residency positions. Some have argued that reporting results as pass/fail would honor the test's purpose while preventing inappropriate use. To date, the USMLE's sponsor organizations have declined to make such a change. In this Perspective, the authors examine the history and mission of the National Board of Medical Examiners (NBME), trace the rise of "Step 1 mania," and consider the current financial incentives for the NBME in implementing a pass/fail score-reporting policy.The NBME was founded in 1915 to address the lack of interstate reciprocity in medical licensure examination. With the creation of the USMLE in 1992, a single pathway for licensure was established, and the organization's original mission was achieved. Yet even after fulfilling its primary purpose, the NBME-classified as a nonprofit organization-has seen its revenues rise dramatically over the past 2 decades. Much of the increased revenue is derived from test products and services not required for medical licensure, with sales driven by the increasing importance of Step 1 scores in residency selection. Revenue from these products and services would likely decline if the NBME reported Step 1 results as pass/fail.A financial conflict of interest occurs when a judgment concerning a primary interest may be influenced by a secondary interest, such as financial gain. The data presented here demonstrate that the NBME has a conflict of interest in its current score-reporting policy. Possible remedies, such as disclosure, recusal, divestiture, and restructuring, are considered.

Online Forums and the "Step 1 Climate": Perspectives From a Medical Student Reddit User

Reddit is a popular content aggregator and discussion website that plays an important role in shaping medical student culture and study habits. The forum r/medicalschool, in particular, provides a distilled view into contemporary U.S. medical students' attitudes and deteriorating relationship to their home institutions' educators and curricula. As a national discussion on United States Medical Licensing Examination Step 1 reform emerges, the role of forums like r/medicalschool in shaping a "Step 1 climate" via the design and dissemination of prescriptive Step 1 study regimens based on commercially available resources and crowdsourced flash card decks goes largely unseen and undiscussed by medical educators. This Invited Commentary aims to introduce medical educators to these forums, highlight the common attitudes borne out of them, and contextualize one popular proposal for Step 1 reform-shifting to a pass/fail exam-within the author's experience as an online forum insider.

Pass/Fail Scoring of USMLE Step 1 and the Need for Residency Selection Reform

The Federation of State Medical Boards and the National Board of Medical Examiners recently announced a change in the United States Medical Licensing Examination Step 1 scoring convention to take effect, at the earliest, on January 1, 2022. There are many reasons for this change, including decreasing medical student stress and incentivizing students to learn freely without solely focusing on Step 1 performance. The question remains how this will affect the future of the otolaryngology-head and neck surgery match. By eradicating Step 1 grades, other factors, such as research, may garner increased importance in the application process. Such a shift may discriminate against students from less well-known medical schools, international medical graduates, and students from low socioeconomic backgrounds, who have fewer academic resources and access to research. Residency programs should try to anticipate such unintended consequences of the change and work on solutions heading into 2022.

Evaluating Urology Residency Applications: What Matters Most and What Comes Next?

PURPOSE OF REVIEW

In light of the announcement that the United States Medical Licensing Examination Step 1 exam will transition to pass/fail reporting, we reviewed recent literature on evaluating residency applicants with a focus on identifying objective measurements of applicant potential.

RECENT FINDINGS

References from attending urologists, Step 1 scores, overall academic performance, and research publications are among the most important criteria used to assess applicants. There has been a substantial increase in the average number of applications submitted per applicant, with both applicants and residency directors indicating support for a cap on the number of applications that may be submitted. Additionally, there are increasing efforts to promote diversity with the goal of improving care and representation in urology. Despite progress in standardizing interview protocols, inappropriate questioning remains an issue. Opportunities to improve residency application include promoting diversity, enforcing prohibitions of illegal practices, limiting application numbers, and finding more transparent and equitable screening measures to replace Step 1.

Blackface, Implicit Bias, and the Informal Curriculum: Shaping the Healthcare Workforce, and Improving Health

Blackface, Implicit Bias, and the Informal Curriculum: Shaping the Healthcare Workforce, and Improving Health.

BACKGROUND/PURPOSE

Health disparities disproportionately affect minority populations and are due to multiple factors including discrimination and implicit bias. Biases are ubiquitous throughout society, including the educational and healthcare environment. In health care it is at the patient-provider level that our biases play a major role in patient care exhibiting a rippling effect going beyond individual provider biases affecting not only patients and families but all members of the healthcare team.

METHODS

Although biases are in play across the entire medical school curriculum the most significant impact is during clerkships. During clerkships students are exposed to and prone to adopt and internalize identities and traits that may run counter to the basic tenants of medicine and the Hippocratic tradition of non-maleficence, beneficence, and compassion. Implicit biases develop early, are difficult to change and as shown by recent allegations of political figures appearing in blackface remain intact into adulthood.

CONCLUSION/RECOMMENDATIONS

At the institutional level biases can be addressed and mitigated through cultural humility and implicit bias training, training in culturally and linguistically appropriate services, increased workforce diversity through support of STEM- based enrichment programs and curriculum changes that include clinical vignettes emphasizing the effects of race, gender, sexual orientation, and ethnicity, on access and quality of care; reflective writing and small group sessions that provide safe spaces for students; and experiential learning in underserved communities. Resulting in an educational environment that directly addresses the role of implicit bias, racism, and discrimination in individual and population health.

Beyond High-Stakes Testing: Learner Trust, Educational Commodification, and the Loss of Medical School Professionalism

With ever-growing emphasis on high-stakes testing in medical education, such as the Medical College Admission Test and the United States Medical Licensing Examination Step 1, there has been a recent surge of concerns on the rise of a "Step 1 climate" within U.S. medical schools. The authors propose an alternative source of the "climate problem" in current institutions of medical education. Drawing on the intertwined concepts of trust and professionalism as organizational constructs, the authors propose that the core problem is not hijacking-by-exam but rather a hijackable learning environment weakened by a pernicious and under-recognized tide of commodification within the U.S. medical education system. The authors discuss several factors contributing to this weakening of medicine's control over its learning environments, including erosion of trust in medical school curricula as adequate preparation for entry into the profession, increasing reliance on external profit-driven sources of medical education, and the emergence of an internal medical education marketplace. They call attention to breaches in the core tenets of a profession-namely a logic that differentiates its work from market and managerial forces, along with related slippages in discretionary decision making. The authors suggest reducing reliance on external performance metrics (high-stakes exams and corporate rankings), identifying and investing in alternative metrics that matter, abandoning the marketization of medical education "products," and attending to the language of educational praxis and its potential corruption by market and managerial lexicons. These steps might salvage some self-governing independence implied in the term "profession" and make possible (if not probable) a recovery of a public trust becoming of the term and its training institutions.

Can Better Selection Tools Help Us Achieve Our Diversity Goals in Postgraduate Medical Education? Comparing Use of USMLE Step 1 Scores and Situational Judgment Tests at 7 Surgical Residencies

PURPOSE

Use of the United States Medical Licensing Examination (USMLE) for residency selection has been criticized for its inability to predict clinical performance and potential bias against underrepresented minorities (URMs). This study explored the impact of altering traditional USMLE cutoffs and adopting more evidence-based applicant screening tools on inclusion of URMs in the surgical residency selection process.

METHOD

Multimethod job analyses were conducted at 7 U.S. general surgical residency programs during the 2018-2019 application cycle to gather validity evidence for developing selection assessments. Unique situational judgment tests (SJTs) and scoring algorithms were created to assess applicant competencies and fit. Programs lowered their traditional USMLE Step 1 cutoffs and invited candidates to take their unique SJT. URM status (woman, racial/ethnic minority) of candidates who would have been considered for interview using traditional USMLE Step 1 cutoffs was compared with the candidate pool considered based on SJT performance.

RESULTS

A total of 2,742 general surgery applicants were invited to take an online SJT by at least 1 of the 7 programs. Approximately 35% of applicants who were invited to take the SJT would not have met traditional USMLE Step 1 cutoffs. Comparison of USMLE-driven versus SJT-driven assessment results demonstrated statistically different percentages of URMs recommended, and including the SJT allowed an average of 8% more URMs offered an interview invitation (P < .01).

CONCLUSIONS

Reliance on USMLE Step 1 as a primary screening tool precludes URMs from being considered for residency positions at higher rate than non-URMs. Developing screening tools to measure a wider array of candidate competencies can help create a more equitable surgical workforce.

Investigating the Relationship Between a Clinical Science Composite Score and USMLE Step 2 Clinical Knowledge and Step 3 Performance

Previous research has found a moderate relationship between performance on individual clinical science subject examinations and USMLE performance. Given the widespread use of the clinical science subject examinations and the need for measures of clinical knowledge that help predict performance on Steps 2 CK and 3 and performance in residency training, this study explores the use of composite scores based on clinical science subject examinations to predict clinical knowledge outcome measures. The data set included students who took all of the five most widely used clinical science subject examinations (medicine, obstetrics and gynecology, pediatrics, psychiatry, and surgery) between January 1, 2013, and December 31, 2017 (N = 65,516). Composite scores were calculated based on average equated percent correct scores across various combinations of clinical science subject examinations. Stepwise linear regression analyses were performed with composite score and Step 1 score as predictor variables and Step 2 CK score or Step 3 score as the dependent variable. In all cases, the proportion of variance explained (R ² ) by the composite score (0.62-0.65 for Step 2 CK score and 0.45-0.48 for Step 3 score) was greater than R ² for Step 1 by itself (0.52 for Step 2 CK score and 0.37 for Step 3 score). Logistic regression analyses found that higher composite scores were associated with a greater probability of passing Steps 2 CK and 3. Composite scores can be used alone or in conjunction with Step 1 to identify students at risk of failing Step 2 CK and/or Step 3 to facilitate remediation.

The association between United States Medical Licensing Examination scores and clinical performance in medical students

Purpose: United States Medical Licensing Examination (USMLE) Step 1 and Step 2 Clinical Knowledge (CK) scores are frequently used to evaluate applicants to residency programs. Recent literature questions the value of USMLE scores for evaluation of residency applicants, in part due to a lack of evidence supporting a relationship with clinical performance. This study explored the relationship between USMLE scores and medical students' clinical performance, as measured by the count of honors grades received in core clinical clerkships. Methods: USMLE Step 1 and Step 2 CK scores and number of honors grades per student in seven core clinical clerkships were obtained from 1,511 medical students who graduated in 2013-2017 from two medical schools. The relationships between variables were analyzed using correlation coefficients, independent-samples t-tests, and hierarchical multiple regression. Results: Count of honors grades correlated with both Step 1 (R=0.480, P<0.001) and Step 2 CK (R=0.542, P<0.001). After correcting for gender, institution, and test-taking ability (using MCAT scores as a proxy for test-taking ability) in a hierarchical multiple regression model, Step 1 and Step 2 CK scores together explained 22.2% of the variance in count of honors grades. Conclusion: USMLE Step 1 and Step 2 CK scores moderately correlate with the number of honors grades per student in core clinical clerkships. This relationship is maintained even after correcting for gender, institution, and test-taking ability. These results indicate that USMLE scores have a positive linear association with clinical performance as a medical student.

Medical education today: all that glitters is not gold

BACKGROUND

The medical education system based on principles advocated by Flexner and Osler has produced generations of scientifically grounded and clinically skilled physicians whose collective experiences and contributions have served medicine and patients well. Yet sweeping changes launched around the turn of the millennium have constituted a revolution in medical education. In this article, a critique is presented of the new undergraduate medical education (UME) curricula in relationship to graduate medical education (GME) and clinical practice.

DISCUSSION

Medical education has changed and will continue to change in response to scientific advances and societal needs. However, enthusiasm for reform needs to be tempered by a more measured approach to avoid unintended consequences. Movement from novice to master in medicine cannot be rushed. An argument is made for a shoring up of biomedical science in revised curricula with the beneficiaries being nascent practitioners, developing physician-scientists --and the public.

CONCLUSION

Unless there is further modification, the new integrated curricula are at risk of produce graduates deficient in the characteristics that have set physicians apart from other healthcare professionals, namely high-level clinical expertise based on a deep grounding in biomedical science and understanding of the pathologic basis of disease. The challenges for education of the best possible physicians are great but the benefits to medicine and society are enormous.