Association Between Internal Medicine Residency Applicant Characteristics and Performance on ACGME Milestones During Intern Year

A Systematic Review of Metrics Utilized in the Selection and Prediction of Future Performance of Residents in the United States

Background Aligning resident and training program attributes is critical. Many programs screen and select residents using assessment tools not grounded in available evidence. This can introduce bias and inappropriate trainee recruitment. Prior reviews of this literature did not include the important lens of diversity, equity, and inclusion (DEI). Objective This study's objective is to summarize the evidence linking elements in the Electronic Residency Application Service (ERAS) application with selection and training outcomes, including DEI factors. Methods A systematic review was conducted on March 30, 2022, concordant with PRISMA guidelines, to identify the data supporting the use of elements contained in ERAS and interviews for residency training programs in the United States. Studies were coded into the topics of research, awards, United States Medical Licensing Examination (USMLE) scores, personal statement, letters of recommendation, medical school transcripts, work and volunteer experiences, medical school demographics, DEI, and presence of additional degrees, as well as the interview. Results The 2599 identified unique studies were reviewed by 2 authors with conflicts adjudicated by a third. Ultimately, 231 meeting inclusion criteria were included (kappa=0.53). Conclusions Based on the studies reviewed, low-quality research supports use of the interview, Medical Student Performance Evaluation, personal statement, research productivity, prior experience, and letters of recommendation in resident selection, while USMLE scores, grades, national ranking, attainment of additional degrees, and receipt of awards should have a limited role in this process.

Postgraduate Selection in Medical Education: A Scoping Review of Current Priorities and Values

PURPOSE

The process of screening and selecting trainees for postgraduate training has evolved significantly in recent years, yet remains a daunting task. Postgraduate training directors seek ways to feasibly and defensibly select candidates, which has resulted in an explosion of literature seeking to identify root causes for the problems observed in postgraduate selection and generate viable solutions. The authors therefore conducted a scoping review to analyze the problems and priorities presented within the postgraduate selection literature to explore practical implications and present a research agenda.

METHOD

Between May 2021 and February 2022, the authors searched PubMed, EMBASE, Web of Science, ERIC, and Google Scholar for English language literature published after 2000. Articles that described postgraduate selection were eligible for inclusion. 2,273 articles were ultimately eligible for inclusion. Thematic analysis was performed on a subset of 100 articles examining priorities and problems within postgraduate selection. Articles were sampled to ensure broad thematic and geographical variation across the breadth of articles that were eligible for inclusion.

RESULTS

Five distinct perspectives or value statements were identified in the thematic analysis: (1) Using available metrics to predict performance in postgraduate training; (2) identifying the best applicants via competitive comparison; (3) seeking alignment between applicant and program in the selection process; (4) ensuring diversity, mitigation of bias, and equity in the selection process; and (5) optimizing the logistics or mechanics of the selection process.

CONCLUSIONS

This review provides insight into the framing and value statements authors use to describe postgraduate selection within the literature. The identified value statements provide a window into the assumptions and subsequent implications of viewing postgraduate selection through each of these lenses. Future research must consider the outcomes and consequences of the value statement chosen and the impact on current and future approaches to postgraduate selection.

Association of Interview and Holistic Review Metrics With Resident Performance-Related Difficulties in an Internal Medicine Residency Program

Background The utility of traditional academic factors to predict residency candidates' performance is unclear. Many programs utilize holistic review processes assessing applicants on an expanded range of application and interview characteristics. Determining which characteristics might predict performance-related difficulty in residency is needed. Objective We aim to elucidate factors associated with residency performance-related difficulty in a large academic internal medicine residency program. Methods In 2022, we conducted a retrospective cohort study of Electronic Residency Application Service and interview data for residents matriculating between 2018 and 2020. The primary outcome was a composite of performance-related difficulty during residency (referral to the Clinical Competency Committee; any rotation evaluation score of 2 out of 5 or lower; and/or a confidential "comment of concern" to the program director). Logistic regression models were fit to assess associations between resident characteristics and the composite outcome. Results Thirty-eight of 117 residents met the composite outcome. Gold Humanism Honor Society (odds ratio [OR] 0.24, 95% confidence interval [CI] 0.16-0.87) or Alpha Omega Alpha (OR 0.36, 95% CI 0.14-0.99) members were less likely to have performance-related difficulty, as were residents with higher United States Medical Licensing Examination Step 2 Clinical Knowledge scores (OR 0.97, 95% CI 0.47-1.00). One-point increases in general faculty overall interview score, leadership competency score, and leadership overall score were associated with 41% to 63% lower odds of meeting the composite outcome. Interview or file review "flags" had an OR of 2.82 (95% CI 1.37-5.80) for the composite outcome. Conclusions Seven metrics were associated with the composite outcome of resident performance-related difficulty.

Are Final Residency Milestones Correlated With Early Fellowship Performance in Pediatrics?

PURPOSE

Milestones have been used to assess trainees across graduate medical education programs and reflect a developmental continuum from novice to expert. This study examined whether residency milestones are correlated with initial fellowship milestone performance in pediatrics.

METHOD

This retrospective cohort study used descriptive statistics to assess milestone scores from pediatric fellows who began fellowship training between July 2017 and July 2020. Milestone scores were obtained at the end of residency (R), middle of the first fellowship year (F1), and end of the first fellowship year (F2).

RESULTS

Data represent 3,592 unique trainees. High composite R scores, much lower F1 scores, and slightly higher F2 scores were found over time for all pediatric subspecialities. R scores were positively correlated with F1 scores (Spearman ρ = 0.12, P < .001) and F2 scores (Spearman ρ = 0.15, P < .001). Although scores are negligibly different when trainees graduate from residency, there were differences in F1 and F2 scores among fellows in different specialties. Those who trained at the same institution for residency and fellowship had higher composite milestone F1 and F2 scores compared with those who trained at different institutions ( P < .001). The strongest associations were between R and F2 scores for the professionalism and communication milestones, although associations were still relatively weak overall (r s = 0.13-0.20).

CONCLUSIONS

This study found high R scores and low F1 and F2 scores across all shared milestones, with weak association of scores within competencies, indicating that milestones are context dependent. Although professionalism and communication milestones had a higher correlation compared with the other competencies, the association was still weak. Residency milestones may be useful for individualized education in early fellowship, but fellowship programs should be cautious about overreliance on R scores due to the weak correlation with F1 and F2 scores.

Efforts to Recruit Diverse Resident Complements in Family Medicine

BACKGROUND AND OBJECTIVES

Family medicine is the most demographically diverse specialty in medicine today. Specialty associations and the Accreditation Council for Graduate Medical Education (ACGME) urge residency programs to engage in systematic efforts to recruit diverse resident complements. Using responses from program directors to the ACGME's mandatory annual update, we enumerate the efforts in resident recruiting. This allows us to compare these statements to the recommendations of two highly respected commissions: the Sullivan Commission on Diversity in the Healthcare Workforce and the Institute of Medicine's In the Nation's Compelling Interest: Ensuring Diversity of the Healthcare Workforce.

METHODS

We compiled the annual updates from 689 family medicine programs and analyzed them using a qualitative method called template analysis. We then classified the efforts and compared them to the recommendations of the Sullivan Commission and Institute of Medicine (IOM).

RESULTS

Nearly all (98%) of the programs completed the portion of the annual update inquiring about recruiting residents. The Sullivan Commission and IOM recommended 23 steps to diversify workforce recruiting. We found that programs engaged in all but one of these recommendations. Among the most frequently employed recommendations were doing holistic reviews and using data for planning. None mentioned engaging in public awareness campaigns. Programs also implemented eight strategies not suggested in either report, with staff training in nondiscrimination policies being among the most frequently mentioned. Among program efforts not included in the Sullivan Commission or IOM recommendations were extracurricular activities; appointing diversity, equity, and inclusion (DEI) committees or advocates; subinternship (Sub-I) experiences; recruiting at conferences; blind reviews; legal compliance; and merit criteria. In total, we found 31 interventions in use.

CONCLUSIONS

The Sullivan Commission's guidance, IOM recommendations, and program-developed initiatives can be combined to create a comprehensive roster of diversity recruiting initiatives. Programs may use this authoritative resource for identifying their next steps in advancing their recruiting efforts.

Fixing a Broken Clerkship Assessment Process: Reflections on Objectivity and Equity Following the USMLE Step 1 Change to Pass/Fail

Clerkship grading is a core feature of evaluation for medical students' skills as physicians and is considered by most residency program directors to be an indicator of future performance and success. With the transition of the U.S. Medical Licensing Examination Step 1 score to pass/fail, there will likely be even greater reliance on clerkship grades, which raises several important issues that need to be urgently addressed. This article details the current landscape of clerkship grading and the systemic discrepancies in assessment and allocation of honors. The authors examine not only objectivity and fairness in clerkship grading but also the reliability of clerkship grading in predicting residency performance and the potential benefits and drawbacks to adoption of a pass/fail clinical clerkship grading system. In the promotion of a more fair and equitable residency selection process, there must be standardization of grading systems with consideration of explicit grading criteria, grading committees, and/or structured education of evaluators and assessors regarding implicit bias. In addition, greater adherence and enforcement of transparency in grade distributions in the Medical Student Performance Evaluation is needed. These changes have the potential to level the playing field, foster equitable comparisons, and ultimately add more fairness to the residency selection process.

Values-Based Resident Selection in an Internal Medicine-Pediatrics Residency Program

BACKGROUND

There is no consensus regarding values important for medical resident success, and current methods for selecting residents correlate poorly with success in residency.

OBJECTIVE

We developed and validated a set of values demonstrated by exemplary residents in the Internal Medicine-Pediatrics program at the University of Utah and used them to inform our resident selection process.

DESIGN

We utilized a modified Delphi method to identify and internally validate values of successful residents. We implemented these values into the interview evaluation rubric.

PARTICIPANTS

Four members of the Internal Medicine-Pediatrics residency program leadership and eleven current residents aided in value generation. Nine faculty from leadership positions in the residency programs of Internal Medicine-Pediatrics, Internal Medicine, and Pediatrics formed a local expert panel for validation.

APPROACH

We performed a literature review and engaged local stakeholders in a semi-structured group interview to generate 107 values. After consolidation based on redundancy, two iterative cycles of expert review using a modified Delphi approach, and alignment with the Accreditation Council for Graduate Medical Education core competencies, eleven values achieved expert agreement and were integrated into an interview rubric to aid in resident selection.

KEY RESULTS

We identified eleven values important for resident success: academic strength, intellectual curiosity, compassion, communication, work ethic, teamwork, leadership, self-awareness, DEI (diversity, equity, and inclusion), professionalism, and adaptability. The rank list from 2021 was found to correlate with a score based on values, but not Step 2 score, as it did in 2017.

CONCLUSIONS

We applied a modified Delphi method to generate eleven observable values present in the ideal Internal Medicine-Pediatric resident at one academic health center in the Intermountain West. Higher Step 2 scores no longer correlated with higher ranking when we used these values to inform our rank list.

Accreditation Council for Graduate Medical Education Milestone Scores in Pediatrics: Pilot Study Exploring the Relationship Between Residency and Early Fellowship Scores

BACKGROUND

The Accreditation Council for Graduate Medical Education requires Milestone-based assessments of residents and fellows. In pediatrics, 11 subcompetencies are common to both residency and subspecialty fellowship training. It is unknown whether Milestone scores achieved during residency are related to Milestone scores achieved in early fellowship.

OBJECTIVE

To assess the relationship between final residency Milestones scores and first-year fellowship Milestones scores in the 11 common subcompetencies (CSCs) across pediatric subspecialties.

METHODS

This was a retrospective single-institution cohort study of pediatric fellows beginning fellowship training between July 2016 and July 2019. De-identified Milestone score sets for final residency scores (R), mid-year first-year fellowship scores (F1), and final first-year fellowship scores (F2) were collected. Spearman correlation and regression analyses were used to assess score relationships.

RESULTS

Data for 85 of 98 eligible fellows were collected. Consistently, the F1 scores were lowest, and the R scores were highest. There was a weak positive correlation between the composite R scores and the composite F1 scores. There was a weak positive correlation between residency and fellowship scores for 6 CSCs and no significant correlation for the remaining 5.

CONCLUSION

For the 11 pediatric CSCs, the final residency Milestone scores are consistently higher than and only weakly associated with early fellowship Milestone scores. There may be limitations to the use of residency scores for fellowship program directors in guiding individualized education for early fellows. This study provides groundwork for additional study of Milestone relationships and may help inform the next iteration of pediatric subspecialty Milestones.