The utility of pre-residency standardized tests for anesthesiology resident selection: the place of United States Medical Licensing Examination scores

Resident Recruitment in a New Era

ABSTRACT

This chapter focuses on resident recruitment and recent US National Resident Matching Program changes and the impact in the evaluation and ranking of applicants within the specialty of anesthesiology. Recruitment challenges are examined as well as program strategies and potential future directions. Also discussed are DEI initiatives within the recruitment process.

Student Perspectives on the Undergraduate Medical Student Research Experience at a Single United States Allopathic Institution

Introduction The purpose of this study was to identify student-reported institutional facilitators and barriers to successful research experiences at a single United States allopathic institution. Residency applications have increasingly become more competitive, and with the United States Medical Licensing Examination (USMLE) Step 1 exam's transition to pass/fail, factors such as research experience and outcomes may become more important to increase residency application competitiveness. This study sought to explore factors that impact successful research experiences leading to tangible outcomes for medical students at our medical school, the Joe R. & Teresa Lozano Long School of Medicine. Methods  A cross-sectional survey was developed and administered via REDCap to 853 students in May 2022. Survey question domains included demographics, past and present research participation, perceived barriers/facilitators to research, tangible outcomes (e.g., publications and posters), and overall satisfaction with research comparing subjectively "best" and "worst" experiences. The Institutional Review Board (IRB) deemed this project as non-regulated research.  Results We had a 24% (n = 204/853) response rate. The responses were distributed equally among the four classes. A big portion of the participants (71%, n = 59/83) identified a tangible outcome as the most important measure of success. Regarding facilitators, students identified having a mentor (89%, n = 165/184) and departmental connections (85%, n = 156/184) as the most important when looking for a project. Barriers included SMART goals (Specific, Measurable, Achievable, Relevant, and Time-Bound) lacking in 31% (n = 24/75) of worst projects, followed by a clear timeline in 29% (n = 22/76) and hours of commitment in 27% (n = 21/78). The best projects were more likely to have resulted in a publication (61% (27/44) vs. 32% (14/44)) or have a poster (64% (28/44) vs. 36% (16/44)). Conclusions Medical students are interested in participating in research, with important facilitators including mentorship and departmental connections. Modifiable variables include lack of clear timelines, well-defined roles and responsibilities, and time commitments. This information may be useful for faculty who mentor medical students or medical schools interested in designing medical student research programs.

The Art and Science of Securing an Anaesthesiology Training Job: A Survey of the Key Factors in Two Australian Tertiary Hospitals

Background There is no specific formal guidance on what prospective trainees must focus on to secure an anaesthetic training position in Australia, and there is little in the literature to advise both applicants and their mentors. Method This study aims to ascertain the views of anaesthetic clinicians from two Australian tertiary referral hospitals on what they consider most important for selection. A paper-based survey was conducted at both hospitals across three groups, totalling 104 participants with a 100% response rate. Results The characteristics most agreed upon to be of at least some importance were clinical anaesthetic knowledge (98%, 102/104), teaching (95%, 99/104), basic science and courses (94%, 98/104), other critical care experience (93%, 97/104), and anaesthetic experience for more than six months (92%, 96/104). Of these, anaesthetic experience of greater than six months, non-anaesthetic critical care experience, and the completion of relevant courses were felt to be most important. Furthermore, good referee reports (95%, 99/104), especially those that come from anaesthetists (75%, 78/104) as well as having previous experience working in the institution applied to (88%, 92/104) were also seen as important factors. 'Non-technical' skills (40%, 42/104) were also regarded as an important factor, with immense competition for a few training positions (45%, 47/104) as the greatest barrier. When it came to selection, prevocational trainees consistently ranked the majority of criteria higher than accredited trainees or specialists. Conclusion This staff survey in two Australian hospitals has shed light on factors considered critical in securing an anaesthetic training position. It underscores the significance of clinical anaesthetic knowledge, basic science proficiency, and relevant critical care experience.

The Association Between USMLE Step 2 Clinical Knowledge Scores and Residency Performance: A Systematic Review and Meta-Analysis

PURPOSE

With the change in Step 1 score reporting, Step 2 Clinical Knowledge (CK) may become a pivotal factor in resident selection. This systematic review and meta-analysis seeks to synthesize existing observational studies that assess the relationship between Step 2 CK scores and measures of resident performance.

METHOD

The authors searched MEDLINE, Web of Science, and Scopus databases using terms related to Step 2 CK in 2021. Two researchers identified studies investigating the association between Step 2 CK and measures of resident performance and included studies if they contained a bivariate analysis examining Step 2 CK scores' association with an outcome of interest: in-training examination (ITE) scores, board certification examination scores, select Accreditation Council for Graduate Medical Education core competency assessments, overall resident performance evaluations, or other subjective measures of performance. For outcomes that were investigated by 3 or more studies, pooled effect sizes were estimated with random-effects models.

RESULTS

Among 1,355 potential studies, 68 met inclusion criteria and 43 were able to be pooled. There was a moderate positive correlation between Step 2 CK and ITE scores (0.52, 95% CI 0.45-0.59, P < .01). There was a moderate positive correlation between Step 2 CK and ITE scores for both nonsurgical (0.59, 95% CI 0.51-0.66, P < .01) and surgical specialties (0.41, 95% CI 0.33-0.48, P < .01). There was a very weak positive correlation between Step 2 CK scores and subjective measures of resident performance (0.19, 95% CI 0.13-0.25, P < .01).

CONCLUSIONS

This study found Step 2 CK scores have a statistically significant moderate positive association with future examination scores and a statistically significant weak positive correlation with subjective measures of resident performance. These findings are increasingly relevant as Step 2 CK scores will likely become more important in resident selection.

Interventions Associated With Racial and Ethnic Diversity in US Graduate Medical Education: A Scoping Review

Importance

Racially and ethnically minoritized individuals remain underrepresented in graduate medical education relative to their proportion in the population. While many programs and initiatives have been developed to address this problem, there is little consensus regarding strategies that work to improve representation across specialties.

Objective

To examine and synthesize evidence-based practices that have been used to increase the proportions of underrepresented in medicine (URiM) trainees at US residency and fellowship programs.

Evidence Review

The authors searched PubMed, Google Scholar, Embase, PsycInfo, ERIC, Cochrane Reviews, Cochrane Trials, CINAHL, Scopus, and PROSPERO electronic databases to identify relevant studies published through January 2022. They screened all titles and abstracts for relevance and read full-text articles to identify articles reporting reliable data describing the outcomes of interventions to improve racial and ethnic diversity among trainees.

Findings

Twenty-seven articles were included in this review. Two studies reported on fellowship programs. The most common interventions included holistic review (48%), decreased emphasis on United States Medical Licensing Examination Step 1 scores (48%), and explicit institutional messaging regarding the importance of diversity (37%). A combination of interventions was associated with an increased number of URiM applicants, interviewees, and matriculants across various medical and surgical specialties.

Conclusions and Relevance

In this scoping review, approaches and interventions associated with increased diversity in residency and fellowship programs were identified. Continued efforts are necessary to sustain such efforts and assess long-term outcomes.

Workforce solutions to address health disparities

PURPOSE OF REVIEW

This review focuses on physician workforce racial & ethnic diversity as a solution to improve perioperative and peripartum health equity.

RECENT FINDINGS

Black, Indigenous, and Hispanic physicians remain underrepresented in medicine (URiM) and anesthesiology, and efforts to expand this workforce have had limited impact. Psychological forces, including implicit bias, aversive racism, outgroup bias, racial attention bias, stereotype threat, and imposter syndrome all act to reinforce structural racism and decrease opportunity for advancement. Evidence based solutions are emerging, but require institutional commitment and widespread engagement of the entire medical community.

SUMMARY

Academic medicine has recognized the need to diversify the physician workforce for more than 50 years, and yet Black, Indigenous, and Hispanic physicians remain URiM. Foundational assumptions and power structures in medicine limit entry, advancement, and retention of URiM physicians. Solutions require leadership and institutional commitment to change the policies, procedures, priorities, and culture of academic medicine.

Association Between USMLE Step 1 Scores and In-Training Examination Performance: A Meta-Analysis

PURPOSE

On February 12, 2020, the sponsors of the United States Medical Licensing Examination announced that Step 1 will transition to pass/fail scoring in 2022. Step 1 performance has historically carried substantial weight in the evaluation of residency applicants and as a predictor of subsequent subject-specific medical knowledge. Using a systematic review and meta-analysis, the authors sought to determine the association between Step 1 scores and in-training examination (ITE) performance, which is often used to assess knowledge acquisition during residency.

METHOD

The authors systematically searched Medline, EMBASE, and Web of Science for observational studies published from 1992 through May 10, 2020. Observational studies reporting associations between Step 1 and ITE scores, regardless of medical or surgical specialty, were eligible for inclusion. Pairs of researchers screened all studies, evaluated quality assessment using a modified Newcastle-Ottawa Scale, and extracted data in a standardized fashion. The primary endpoint was the correlation of Step 1 and ITE scores.

RESULTS

Of 1,432 observational studies identified, 49 were systematically reviewed and 37 were included in the meta-analysis. Overall study quality was low to moderate. The pooled estimate of the correlation coefficient was 0.42 (95% confidence interval [CI]: 0.36, 0.48; P < .001), suggesting a weak-to-moderate positive correlation between Step 1 and ITE scores. The random-effects meta-regression found the association between Step 1 and ITE scores was weaker for surgical (versus medical) specialties (beta -0.25 [95% CI: -0.41, -0.09; P = .003]) and fellowship (versus residency) training programs (beta -0.25 [95% CI: -0.47, -0.03; P = .030]).

CONCLUSIONS

The authors identified a weak-to-moderate positive correlation between Step 1 and ITE scores based on a meta-analysis of low-to-moderate quality observational data. With Step 1 scoring transitioning to pass/fail, the undergraduate and graduate medical education communities should continue to develop better tools for evaluating medical students.

Small Animal Resident Selection Processes at a University Teaching Hospital: An Analysis and Recommendations for Improvement

Concerns regarding resident performance within a small animal department prompted a review of selection practices, with the intent of improving validity and efficiency. Information was gathered from semi-structured interviews and descriptions of current processes; emphasis was placed on determining how the Veterinary Internship and Residency Matching Program application was used. Processes were found to lack standardization and rely heavily on arbitrary judgments. In addition, faculty members expressed concerns regarding their reliability and the time spent generating candidate rankings. Suggestions for improvement were based on current practices in personnel psychology and human resource management. The need for standardization within and across specialty groups was emphasized, along with a multiple-hurdle approach in which a substantial deficit or red flag in any component results in candidate disqualification. Comprehensive recommendations were made for the selection process as follows: Each application undergoes initial administrative screening for employment eligibility and academic cut-offs; eligible applications are scored by 2-3 faculty members using defined ratings on four equally weighted pre-interview criteria (i.e., veterinary education, post-graduation experiences, personal statement, and standardized letters of reference); phone calls to colleagues with knowledge of the applicant follow specific guidelines and a rating scale; veterinary-situational structured interview questions with appropriate rating scales are used to assess candidates' standing on specified competencies identified as important for success; and the interview score is weighted equally and added to the four pre-interview components to determine the final rank. It is hoped this new approach will take less time and facilitate the selection of successful residents.

The Change of USMLE Step 1 to Pass/Fail: Perspectives of the Surgery Program Director

OBJECTIVE

This study sought to evaluate the perspectives of surgical program directors regarding the change of USMLE Step 1 to pass/fail grading.

DESIGN

Validated electronic survey.

SETTING

Vanderbilt University Medical Center Department of Plastic Surgery.

PARTICIPANTS

Program directors of all ACMGE-accredited General Surgery, Integrated Vascular Surgery, Integrated Thoracic Surgery, and Integrated Plastic Surgery residency programs.

RESULTS

The overall response rate was 55.5%. Most PDs (78.1%) disagreed with the scoring change. Only 19.6% believe this change will improve medical student well-being. For 63.5% of PDs, medical school pedigree will become more important, and 52.7% believe it will place international medical graduates at a disadvantage. Only 6.2% believe Step 2 CK should also be pass/fail, while 88.7% will increase the weight of Step 2 CK and 88.4% will now require Step 2 CK score submission with the electronic residency application service.

CONCLUSIONS

While well-intentioned, changing USMLE Step 1 to pass/fail may have unintended consequences and may disadvantage certain groups of applicants. The emphasis on Step 1, and resulting test-taking apprehension, will likely shift to Step 2 CK. Proponents of equitable evaluation should direct their efforts toward increasing, not decreasing, the number of objective measures available for student assessment.

Factors Predictive of Performance on USMLE Step 1: Do Commercial Study Aids Improve Scores?

BACKGROUND

The United States Medical Licensing Examination (USMLE) Step 1 was the most cited factor in granting medical student applicants' residency interviews in a 2016 NRMP survey. Medical students utilize numerous commercial resources to achieve the highest score possible. This study sought to examine preparatory resources and their association with USMLE Step 1 performance.

METHODS

In 2016 and 2017, 170 medical students from a public, Midwestern medical school completed a voluntary survey after completing USMLE Step 1. The survey asked about study aids, anticipated performance, test preparation time, and practice exam performance. Actual scores and academic performance (overall percent for years 1 and 2 of medical school) were matched to survey responses. Data was analyzed using descriptive and inferential statistical analyses.

RESULTS

USMLE Step 1 performance was associated with academic performance (r = 0.719, p < 0.01). Post hoc tests indicated a positive difference with the UWorld Qbank and exam performance (F _4,134 = 10.29, p < 0.05). Amount of days dedicated to test preparation was negatively correlated with performance in the 2017 student cohort (r = - 0.304, p < 0.01). The use of other commercially available products and the number of practice tests completed were not correlated with performance.

DISCUSSION

Cumulative pre-clinical grades had the strongest association with performance on the USMLE Step 1. Completing UWorld Qbank at least once was positively related with Step 1 scores. However, the use of other study aids, completing multiple practice exams, and delaying exam dates were not statistically significant. This study indicates performance in the pre-clinical curriculum and increased usage of UWorld are strongly associated with USMLE Step 1 performance.

Evaluating the recruitment process into UK anaesthesia core training: a national data linkage study of doctors' performance at selection and subsequent postgraduate training

PURPOSE OF THE STUDY

To explore which factors increase the likelihood of being deemed appointable to core anaesthesia training in the UK and whether those factors subsequently predict performance in postgraduate training.

STUDY DESIGN

Observational study linking UK medical specialty recruitment data with postgraduate educational performance, as measured by Annual Review of Competence Progression (ARCP) outcomes. Data were available for 2782 trainee doctors recruited to anaesthesia core training from 2012 to 2016 with at least one subsequent ARCP outcome.

RESULTS

Both higher interview and shortlisting scores were independent and statistically significant (p≤0.001) predictors of more satisfactory ARCP outcomes, even after controlling for the influence of postgraduate exam failure. It was noted that a number of background variables (eg, age at application) were independently associated with the odds of being deemed appointable at recruitment. Of these, increasing age and experience were also negative predictors of subsequent ARCP rating. These influences became statistically non-significant once ARCP outcomes associated with exam failure were excluded.

CONCLUSIONS

The predictors of 'appointability' largely also predict subsequent performance in postgraduate training, as indicated by ARCP ratings. This provides evidence for the validity of the selection process. Our results also suggest that greater weight could be applied to shortlisting scores within the overall process of ranking applicants for posts.

The predictive value of pre-recruitment achievement on resident performance in anesthesiology

STUDY OBJECTIVE

Selecting candidates for residency positions is challenging and there is little research on the correlation between commonly used selection criteria and subsequent performance in anesthesiology. This study examined the association between the selection measures and post-recruitment performance in residency.

DESIGN

Retrospective review of archival data.

SETTING

Anesthesiology residency program at a large academic anesthesiology department.

SUBJECTS

Residents who were matched to the anesthesiology program over 9years (graduation classes of 2006 to 2014).

INTERVENTIONS

None.

MEASUREMENTS

The pre-recruitment achievements included a comprehensive list of measures obtained from residents' application portfolios in conjunction with interview performance. The post-recruitment examination outcomes consisted of the in-training examination (ITE) scores in the three clinical anesthesia (CA) years and first-attempt success on the written board certification examination administered by the American Board of Anesthesiology (ABA). Scholarly output during residency was measured by publication record. Clinical performance at the conclusion of residency was independently rated by three faculty members. Bivariate analysis and regression models were conducted to examine association between predictors and outcomes.

MAIN RESULTS

High United States Medical Licensing Examination (USMLE) scores, class rank in medical school and interview performance were predictive of high examination scores in residency and good clinical performance. Class rank appeared to be the best predictor of scholarly publication and pursuing an academic career beyond residency.

CONCLUSIONS

Comparative performance with classmates (i.e., class rank) in medical school appeared to be an effective predictor of overall performance in residency, which warrants more attention in future study. Although interview performance is subject to recruitment team members' interpretation, it is an important measure to include in recruitment decisions.

Orthopaedic fellowship selection criteria: a survey of fellowship directors

BACKGROUND

The pursuit of a fellowship has become increasingly popular over the past several years, with &gt;90% of graduating orthopaedic residents applying for a fellowship position. Despite the ample literature available pertaining to the selection of orthopaedic residents, there is no similar research for the selection of fellows.

METHODS

Four hundred and fifteen of 475 orthopaedic fellowship program directors in the United States were surveyed on selection criteria used to choose fellows. The survey asked fellowship directors to rank the importance of various criteria on a Likert scale of 1 through 5 points (with 1 point denoting most important and 5 points denoting least important).

RESULTS

The overall response rate was 193 (46.5%) of 415 orthopaedic fellowship program directors. The most important criteria in selecting an applicant for an interview were a letter of recommendation from subspecialty faculty (1.38 points), quality of residency program (2.02 points), and a letter of recommendation from the residency program director (2.12 points). The most important criteria in completing the rank order list following the interview were the interview (1.17 points), a letter of recommendation from subspecialty faculty (1.46 points), a letter of recommendation from the residency program director (2.16 points), and expressed interest in program (2.16 points).

Associations between residency selection strategies and doctor performance: a meta-analysis

OBJECTIVES

The purpose of this study was to use meta-analysis to establish which of the information available to the resident selection committee is associated with resident or doctor performance.

METHODS

Multiple electronic databases were searched to 4 September 2012. Two reviewers independently selected studies that met the present inclusion criteria and extracted data in duplicate; disagreement was resolved by consensus. Risk for bias was assessed using a customised bias assessment tool. Measures of association were converted to a common effect size (Hedges' g). Meta-analysis was performed using the random-effects model for each selection strategy and all outcomes without pooling. Sensitivity analysis for each selection strategy-outcome pair was performed with pooling of effect size.

RESULTS

Eighty studies involving a total of 41 704 participants were included in the meta-analysis. Seventeen different selection strategies and 17 outcomes were assessed across these studies. The strongest positive associations referred to examination-based selection strategies, such as the US Medical Licensing Examination (USMLE) Step 1, and examination-based outcomes, such as scores on in-training examinations. Moderate positive associations were present for medical school marks and both examination-based and subjective outcomes. Minimal or no associations were seen for the selection tools represented by interviews, reference letters and deans' letters.

CONCLUSIONS

Standardised examination performance and medical school grades show the strongest associations with current measures of doctor performance. Deans' letters, reference letters and interviews all show a lower than expected strength of association given the relative value often assigned to them during resident doctor selection. Objective selection strategies are potentially the most useful to residency selection committees based on current evaluative methods. However, reports in the literature of validated long-term doctor performance outcomes are scant.

The relationship between the american board of anesthesiology part 1 certification examination and the United States medical licensing examination

BACKGROUND

The graduate medical education community uses results from the United States Medical Licensing Examination (USMLE) to inform decisions about individuals' readiness for postgraduate training.

OBJECTIVE

We sought to determine the relationship between performance on the USMLE and the American Board of Anesthesiology (ABA) Part 1 Certification Examination using a national sample of examinees, and we considered the relationship in the context of undergraduate medical education location and examination content.

METHODS

Approximately 7800 individuals met inclusion criteria. The relationships between USMLE scores and ABA Part 1 pass rates were examined, and predictions for the strength of the relationship between USMLE content areas and ABA performance were compared with observed relationships.

RESULTS

Pearson correlations between ABA Part 1 scores and USMLE Steps 1, 2 (clinical knowledge), and 3 scores for first-taker US/Canadian graduates were .59, .56, and .53, respectively. A clear relationship was demonstrated between USMLE scores and pass rates on ABA Part 1, and content experts were able to successfully predict the USMLE content categories that would least or most likely relate to ABA Part 1 scores.

CONCLUSIONS

The analysis provided evidence on a national scale that results from the USMLE and the ABA Part 1 were correlated and that success on the latter examination was associated with level of USMLE performance. Both testing programs have been successful in conceptualizing many of the knowledge areas of interest and in developing test content to reflect those areas.

The predictive value of general surgery application data for future resident performance

OBJECTIVE

The predictive value of application data for future general surgery resident performance and attrition are poorly understood. We sought to determine what variables obtained in the application process might predict future resident success.

METHODS

We performed an 18-year review (1990-2008) of all matched residents (n = 101) to a university program. Both categorical graduates (CG) and nongraduates (CNG) and nondesignated preliminaries matching (PM) and preliminaries nonmatching (PNM) were evaluated. We also screened for previous high-performance accomplishments outside of the medical field such as in the performing arts or collegiate athletics (SKILL). Outcome data include graduation or match status, American Board of Surgery In-service Training Examination (ABSITE), and faculty Accreditation Council for Graduate Medical Education (ACGME) core competency evaluations.

RESULTS

Background data from the Electronic Residency Application Service (ERAS) application between the various groups was compared with univariate analysis and logistic regression. There were significant differences between the groups on the measures of USMLE step 1 (STEP1) (p = 0.001), medical school grade point average (GPA) (p = 0.023), interview data (INTERVIEW) (p < 0.001), and ABSITE (p < 0.001). The variable of INTERVIEW had an odds ratio of 188.27 (95% confidence interval, 3.757-9435.405). Overall attrition was 23.7% (n = 24) and was evenly divided between those who left for lifestyle reasons and those who were encouraged to leave.

CONCLUSIONS

Within our system, INTERVIEW, USMLE STEP1, and SKILL predict successful completion of a general surgery residency. In contrast to prior reports, female sex, ethnicity, medical school grades, or Alpha Omega Alpha Honor Society (AOA) status were not significant. The variable SKILL is novel and highlights the importance of nonacademic background data. Our data indicate STEP1 is an independent predictor of resident success in general surgery and should maintain an important role in general surgery applicant screening. The ideal screening threshold is likely > 215.