The predictive value of pre-recruitment achievement on resident performance in anesthesiology

Factors associated with infectious diseases fellowship academic success

BACKGROUND

A multitude of factors are considered in an infectious disease (ID) training program's meticulous selection process of ID fellows but their correlation to pre and in-fellowship academic success as well as post-fellowship academic success and short-term outcomes is poorly understood. Our goal was to investigate factors associated with subsequent academic success in fellowship as well as post-fellowship short-term outcomes.

METHODS

In 2022, we retrospectively analyzed deidentified academic records from 39 graduates of the Mayo Clinic Rochester ID Fellowship Program (1 July 2013- 30 June 2022). Data abstracted included demographics, degrees, honor society membership, visa/citizenship status, medical school, residency training program, United States Medical Licensure Exam (USMLE) scores, letters of recommendation, in-training examination (ITE) scores, fellowship track, academic rank, career choice, number of honors, awards, and abstracts/publications prior to fellowship, during training, and within 2 years of graduation.

RESULTS

Younger fellows had higher USMLE step 1 scores, pre and in-fellowship scholarly productivity, and higher ITE performance. Female fellows had significantly higher USMLE step 3 scores. Prior research experience translated to greater in-fellowship scholarly productivity. Higher USMLE scores were associated with higher ID ITE performance during multiple years of fellowship, but USMLE step 2 clinical knowledge and 3 scores were associated with higher pre and in-fellowship scholarly productivity and receiving an award during fellowship. The USMLE step 1 score did not correlate with fellowship performance beyond year 1 and 2 ITE scores.

CONCLUSIONS

Multiple aspects of a prospective fellow's application must be considered as part of a holistic review process for fellowship selection. USMLE step 2 CK and 3 scores may predict fellowship performance across multiple domains.

Subtyping of Performance Trajectory During Medical School, Medical Internship, and the First Year of Residency in Training Physicians: A Longitudinal Cohort Study

BACKGROUND

Developmental trajectories of clinical skills in training physicians vary among tasks and show interindividual differences. This study examined the predictors of medical internship performance and residency entrance and found subtypes of performance trajectory in training physicians.

METHODS

This retrospective cohort study involved 888 training physicians who completed a medical internship between 2015 and 2019. After the internship, 627 physicians applied for residency training between 2016 and 2020. Finally, 160 of them completed their first-year residency in internal medicine, surgery, pediatrics, and psychiatry departments between 2016 and 2020. Pearson's correlation coefficients of internship performance and first year-residency performance (n = 160) were calculated. Latent profile analysis identified performance trajectory subtypes according to medical school grade point average (GPA), internship performance, English proficiency, and residency selection procedures. Multivariate logistic regression models of residency acceptance (n = 627) and performance in the top 30%/lower 10% in the first year of residency were also constructed.

RESULTS

Medical internship performance showed a significant positive correlation with the medical school GPA (r = 0.194) and the written score for the medical licensing examination (r = 0.125). Higher scores in the interview (adjusted odds ratio [aOR], 2.57) and written examination (aOR, 1.45) of residency selection procedures and higher medical internship performance (aOR, 1.19) were associated with a higher chance of residency acceptance. The latent profile analyses identified three training physician subgroups: average performance, consistently high performance (top 30%), and adaptation to changes (lowest 10%). Higher scores in the interview for residency selection (aOR, 1.35) and lower scores for medical internship performance (aOR, 0.79) were associated with a higher chance of performing in the top 30% or lowest 10% in the first year of residency, respectively.

CONCLUSION

Performance in the interview and medical internship predicted being among the top 30% and lowest 10% of performers in the first year of residency training, respectively. Individualized educational programs to enhance the prospect of trainees becoming high-functioning physicians are needed.

Resident Recruitment in a New Era

ABSTRACT

This chapter focuses on resident recruitment and recent US National Resident Matching Program changes and the impact in the evaluation and ranking of applicants within the specialty of anesthesiology. Recruitment challenges are examined as well as program strategies and potential future directions. Also discussed are DEI initiatives within the recruitment process.

The Art and Science of Securing an Anaesthesiology Training Job: A Survey of the Key Factors in Two Australian Tertiary Hospitals

Background There is no specific formal guidance on what prospective trainees must focus on to secure an anaesthetic training position in Australia, and there is little in the literature to advise both applicants and their mentors. Method This study aims to ascertain the views of anaesthetic clinicians from two Australian tertiary referral hospitals on what they consider most important for selection. A paper-based survey was conducted at both hospitals across three groups, totalling 104 participants with a 100% response rate. Results The characteristics most agreed upon to be of at least some importance were clinical anaesthetic knowledge (98%, 102/104), teaching (95%, 99/104), basic science and courses (94%, 98/104), other critical care experience (93%, 97/104), and anaesthetic experience for more than six months (92%, 96/104). Of these, anaesthetic experience of greater than six months, non-anaesthetic critical care experience, and the completion of relevant courses were felt to be most important. Furthermore, good referee reports (95%, 99/104), especially those that come from anaesthetists (75%, 78/104) as well as having previous experience working in the institution applied to (88%, 92/104) were also seen as important factors. 'Non-technical' skills (40%, 42/104) were also regarded as an important factor, with immense competition for a few training positions (45%, 47/104) as the greatest barrier. When it came to selection, prevocational trainees consistently ranked the majority of criteria higher than accredited trainees or specialists. Conclusion This staff survey in two Australian hospitals has shed light on factors considered critical in securing an anaesthetic training position. It underscores the significance of clinical anaesthetic knowledge, basic science proficiency, and relevant critical care experience.

Postgraduate Selection in Medical Education: A Scoping Review of Current Priorities and Values

PURPOSE

The process of screening and selecting trainees for postgraduate training has evolved significantly in recent years, yet remains a daunting task. Postgraduate training directors seek ways to feasibly and defensibly select candidates, which has resulted in an explosion of literature seeking to identify root causes for the problems observed in postgraduate selection and generate viable solutions. The authors therefore conducted a scoping review to analyze the problems and priorities presented within the postgraduate selection literature to explore practical implications and present a research agenda.

METHOD

Between May 2021 and February 2022, the authors searched PubMed, EMBASE, Web of Science, ERIC, and Google Scholar for English language literature published after 2000. Articles that described postgraduate selection were eligible for inclusion. 2,273 articles were ultimately eligible for inclusion. Thematic analysis was performed on a subset of 100 articles examining priorities and problems within postgraduate selection. Articles were sampled to ensure broad thematic and geographical variation across the breadth of articles that were eligible for inclusion.

RESULTS

Five distinct perspectives or value statements were identified in the thematic analysis: (1) Using available metrics to predict performance in postgraduate training; (2) identifying the best applicants via competitive comparison; (3) seeking alignment between applicant and program in the selection process; (4) ensuring diversity, mitigation of bias, and equity in the selection process; and (5) optimizing the logistics or mechanics of the selection process.

CONCLUSIONS

This review provides insight into the framing and value statements authors use to describe postgraduate selection within the literature. The identified value statements provide a window into the assumptions and subsequent implications of viewing postgraduate selection through each of these lenses. Future research must consider the outcomes and consequences of the value statement chosen and the impact on current and future approaches to postgraduate selection.

Factors Associated with Infectious Diseases Fellowship Academic Success

Background

A multitude of factors are considered in an infectious diseases (ID) training program's meticulous selection process of ID fellows but their correlation to pre and in-fellowship academic success as well as post-fellowship academic success and short-term outcomes is poorly understood. Our goal was to investigate factors associated with subsequent academic success in fellowship as well as post-fellowship short-term outcomes.

Methods

In 2022, we retrospectively analyzed deidentified academic records from 39 graduates of the Mayo Clinic Rochester ID Fellowship Program (July 1, 2013- June 30, 2022). Data abstracted included demographics, degrees, honor society membership, visa/citizenship status, medical school, residency training program, United States Medical Licensure Exam (USMLE) scores, letters of recommendation, in-training examination (ITE) scores, fellowship track, academic rank, career choice, number of honors, awards, and abstracts/publications prior to fellowship, during training, and within 2 years of graduation.

Results

Younger fellows had higher USMLE step 1 scores, pre and in-fellowship scholarly productivity, and higher ITE performance. Female fellows had significantly higher USMLE step 3 scores. Prior research experience translated to greater in-fellowship scholarly productivity. Higher USMLE scores were associated with higher ID ITE performance during multiple years of fellowship, but USMLE step 2 clinical knowledge and 3 scores were associated with higher pre and in-fellowship scholarly productivity and receiving an award during fellowship. USMLE step 1 score did not correlate with fellowship performance beyond year 1 and 2 ITE scores.

Conclusions

Multiple aspects of a prospective fellow's application must be considered as part of a holistic reviewprocess for fellowship selection. USMLE step 2 CK and 3 scores may predict fellowship performance across multiple domains.

Medical School Clinical Knowledge Exam Scores, Not Demographic or Other Factors, Associated With Residency In-Training Exam Performance

BACKGROUND

The anesthesiology in-training exam (ITE) is a 200-item multiple-choice assessment completed annually by physician residents. Because all matriculated U.S. Department of Defense (DoD) anesthesiology residents are "hired" by the DoD after residency graduation, it is important to ensure that ITE performance, as a proxy for core competencies achievement, is maximized.

METHODS

Graduated resident program files from 2013 to 2020 were queried for age, sex, matriculant status (medical student vs. other), medical school (Uniformed Services University vs. other), military service (Army vs. Air Force), preresidency military service (yes vs. no), U.S. Medical Licensing Exam (USMLE) Step 2 Clinical Knowledge (CK) score, and the American Board of Anesthesiologists ITE Score from the third clinical anesthesia year (CA-3 year).

RESULTS

For every 1-point increase in USMLE Step 2 CK true z-score, the CA-3 ITE z-score increased by 0.59 points. Age was not associated with CA-3 ITE z-score in any dataset regression. Categorical covariates of sex, application status, medical school, service, and preresidency military service were not significantly associated with CA-3 ITE z-score (all P >.05), as shown by estimated adjusted marginal means. The estimated adjusted grand mean of CA-3 ITE z-scores was 0.48 (standard error ± 0.14).

CONCLUSION

Resident physicians enter residency with varying degrees of past academic success, and it is important to develop early strategies to support them in acquiring the requisite knowledge base.

Metrics of Resident Achievement for Defining Program Aims

Introduction

Resident achievement data is a powerful but underutilized means of program evaluation, allowing programs to empirically measure whether they are meeting their program aims, facilitate refinement of curricula and improve resident recruitment efforts. The goal was to provide an overview of available metrics of resident achievement and how these metrics can be used to inform program aims.

Methods

A literature search was performed using PubMed and Google Scholar between May and November of 2020. Publications were eligible for inclusion if they discussed or assessed “excellence” or “success” during residency training. A narrative review structure was chosen due to the intention to provide an examination of the literature on available resident achievement metrics.

Results

57 publications met inclusion criteria and were included in the review. Metrics of excellence were grouped into larger categories, including success defined by program factors, academics, national competencies, employer factors, and possible new metrics.

Conclusions

Programs can best evaluate whether they are meeting their program aims by creating a list of important resident-level metrics based on their stated goals and values using one or more of the published definitions as a foundation. Each program must define which metrics align best with their individual program aims and mission.

Association Between USMLE Step 1 Scores and In-Training Examination Performance: A Meta-Analysis

PURPOSE

On February 12, 2020, the sponsors of the United States Medical Licensing Examination announced that Step 1 will transition to pass/fail scoring in 2022. Step 1 performance has historically carried substantial weight in the evaluation of residency applicants and as a predictor of subsequent subject-specific medical knowledge. Using a systematic review and meta-analysis, the authors sought to determine the association between Step 1 scores and in-training examination (ITE) performance, which is often used to assess knowledge acquisition during residency.

METHOD

The authors systematically searched Medline, EMBASE, and Web of Science for observational studies published from 1992 through May 10, 2020. Observational studies reporting associations between Step 1 and ITE scores, regardless of medical or surgical specialty, were eligible for inclusion. Pairs of researchers screened all studies, evaluated quality assessment using a modified Newcastle-Ottawa Scale, and extracted data in a standardized fashion. The primary endpoint was the correlation of Step 1 and ITE scores.

RESULTS

Of 1,432 observational studies identified, 49 were systematically reviewed and 37 were included in the meta-analysis. Overall study quality was low to moderate. The pooled estimate of the correlation coefficient was 0.42 (95% confidence interval [CI]: 0.36, 0.48; P < .001), suggesting a weak-to-moderate positive correlation between Step 1 and ITE scores. The random-effects meta-regression found the association between Step 1 and ITE scores was weaker for surgical (versus medical) specialties (beta -0.25 [95% CI: -0.41, -0.09; P = .003]) and fellowship (versus residency) training programs (beta -0.25 [95% CI: -0.47, -0.03; P = .030]).

CONCLUSIONS

The authors identified a weak-to-moderate positive correlation between Step 1 and ITE scores based on a meta-analysis of low-to-moderate quality observational data. With Step 1 scoring transitioning to pass/fail, the undergraduate and graduate medical education communities should continue to develop better tools for evaluating medical students.

Correlation of Ophthalmology Residency Application Characteristics with Subsequent Performance in Residency

Purpose Only from reviewing applications, it is difficult to identify which applicants will be successful ophthalmology residents. The change of the USMLE Step 1 scoring to “Pass/Fail” removes another quantitative metric. We aimed to identify application attributes correlated with successful residency performance. This study also used artificial intelligence (AI) to evaluate letters of recommendation (LOR), the Dean's letter (MSPE), and personal statement (PS).

Design Retrospective analysis of application characteristics versus residency performance was conducted.

Participants Residents who graduated from the Dean McGee Eye Institute/University of Oklahoma Ophthalmology residency from 2004 to 2019 were included in this study.

Methods Thirty-four attributes were recorded from each application. Residents were subjectively ranked into tertiles and top and bottom deciles based on residency performance by faculty present during their training. The Ophthalmic Knowledge Assessment Program (OKAP) examination scores were used as an objective performance metric. Analysis was performed to identify associations between application attributes and tertile/decile ranking. Additional analysis used AI and natural language processing to evaluate applicant LORs, MSPE, and PS.

Main Outcome Measures Characteristics from residency applications that correlate with resident performance were the primary outcome of this study.

Results Fifty-five residents and 21 faculty members were included. A grade of “A” or “Honors” in the obstetrics/gynecology (OB/GYN) clerkship and the presence of a home ophthalmology department were associated with ranking in the top tertile but not the top decile. Mean core clerkship grades, medical school ranking in the top 25 U.S. News and World Report (USNWR) primary care rankings, and postgraduate year (PGY)-2 and PGY-3 OKAP scores were predictive of being ranked in both the top tertile and the top decile. USMLE scores, alpha-omega-alpha (AOA) status, and number of publications did not correlate with subjective resident performance. AI analysis of LORs, MSPE, and PS did not identify any text features that correlated with resident performance.

Conclusions Many metrics traditionally felt to be predictive of residency success (USMLE scores, AOA status, and research) did not predict resident success in our study. We did confirm the importance of core clerkship grades and medical school ranking. Objective measures of success such as PGY-2 and PGY-3 OKAP scores were associated with high subjective ranking.

Utilization of a New Customizable Scoring Tool to Recruit and Select Pulmonary/Critical Care Fellows

Background: Finding the ideal candidate for a residency/fellowship program has always been difficult. Finding the “perfect” match has always been the ultimate goal. However, many factors affect obtaining that “perfect” match. In the past, we would have each attending physician review around 20 to 50 Electronic Residency Application Service (ERAS) applications and rank them into three categories: high, middle, or low. Depending on their ranking, the applicant would be invited for an interview. After the interview, the applicants’ files (ERAS and interview) would be reviewed and ranked by the faculty as a group. This was time-consuming and fraught with too much subjectivity and minimal objectivity. We, therefore, sought to find a way to assess and rank applicants in a more objective and less time-consuming manner. By creating a customizable scoring tool, we were able to screen applicants to our pulmonary/critical care fellowship program in an efficient and a more objective manner.

Objectives: A customizable scoring tool was developed weighting components in the ERAS and interview process, allowing residency/fellowship programs to create a final rank list consistent with the programs’ desired applicants.

Methods: Two hundred and sixty pulmonary/critical care fellowship applications were reviewed from 2013 to 2018. In 2018, we used our new scoring rubric to create a rank list and rescore previous applicants. The traditional and new lists were compared to the final rank list submitted to the National Residency Matching Program (NRMP) for 2018. We wanted to ascertain which scoring method correlated best with the final rank list submitted to the NRMP. We obtained feedback from eight faculty members who had reviewed applicants with both scoring tools.

Results: The novel customizable scoring tool positively correlated with the final rank list submitted to the NRMP (r= 0.86). The novel tool showed a better correlation to the final rank list than the traditional method. Faculties (6/6, 100%) responded positively to the new tool.

Conclusions: Our new customizable tool has allowed us to create a final rank list that is efficient and more focused on our faculty’s desired applicants. We hope to assess and compare the quality of applicants matched through this scoring system and the traditional method by using faculty evaluations, milestones, and test scores.

Educational and Personal Opportunity Costs of Medical Student Preparation for the United States Medical Licensing Examination Step 1 Exam: A Single-Center Study

Purpose

To assess the degree to which medical students choose to disengage from their regular preclinical curriculum and extracurricular activities in order to focus on United States Medical Licensing Examination (USMLE) Step 1 exam preparation, as well as learner-perceived effects of Step 1 preparation on their physical, social, and mental health.

Method

Online survey of medical students who have taken the USMLE Step 1 exam at a single large Midwestern academic medical center.

Results

The response rate was 54%. Students often reported absenteeism from a variety of preclinical curricular activities, including lectures (44%) and didactics focusing on medical ethics (37%), clinical skills (28%), and encounters with actual and standardized patients (9%) in order to study for USMLE Step 1. Many students also forewent extracurricular opportunities including research (53%), elective patient care opportunities (45%), community service (39%), and healthcare advocacy experiences (38%) in order to study for USMLE Step 1. Majorities of students identified Step 1 preparation as a cause of burnout (79%) or significant anxiety or depression (61%), for which nearly a third sought mental healthcare; students also reported Step 1 preparation as a cause of engaging in dangerous behaviors such as illicit prescription stimulant use as well as driving or providing patient care while impaired by fatigue. In narrative comments, students frequently described Step 1 to be a barrier to their development into effective clinicians, the traditional medical school curriculum to be a barrier to performance on Step 1, or both.

Conclusions

Medical students often prioritize Step 1 exam preparation over engaging with the standard preclinical curriculum, extracurricular opportunities, and activities to promote wellbeing. These findings have implications for the emphasis residency program directors place on single high-stakes standardized exams in the resident recruitment process.

Application Factors Associated With Clinical Performance During Pediatric Internship

OBJECTIVE

Our goal was to identify aspects of residency applications predictive of subsequent performance during pediatric internship.

METHODS

We conducted a retrospective cohort study of graduates of US medical schools who began pediatric internship in a large pediatric residency program in the summers of 2013 to 2017. The primary outcome was the weighted average of subjects' Accreditation Council for Graduate Medical Education pediatric Milestone scores at the end of pediatric internship. To determine factors independently associated with performance, we conducted multivariate linear mixed-effects models controlling for match year and Milestone grading committee as random effects and the following application factors as fixed effects: letter of recommendation strength, clerkship grades, medical school reputation, master's or PhD degrees, gender, US Medical Licensing Examination Step 1 score, Alpha Omega Alpha membership, private medical school, and interview score.

RESULTS

Our study population included 195 interns. In multivariate analyses, the aspects of applications significantly associated with composite Milestone scores at the end of internship were letter of recommendation strength (estimate 0.09, 95% confidence intervals [CI]: 0.04, 0.15), numbers of clerkship honors (est. 0.05, 95% CI: 0.01-0.09), medical school ranking (est. 0.04, 95% CI: 0.08-0.01), having a master's degree (est. 0.19, 95% CI: 0.03-0.36), and not having a PhD (est. 0.14, 95% CI: 0.02-0.26). Overall, the final model explained 18% of the variance in milestone scoring.

CONCLUSIONS

Letter of recommendation strength, clerkship grades, medical school ranking, and having obtained a Master's degree were significantly associated with higher clinical performance during pediatric internship.

Characteristics of internal medicine residents who successfully match into cardiology fellowships

BACKGROUND

The unique traits of residents who matriculate into subspecialty fellowships are poorly understood. We sought to identify characteristics of internal medicine (IM) residents who match into cardiovascular (CV) fellowships.

METHODS

We conducted a retrospective cohort study of 8 classes of IM residents who matriculated into residency from 2007 to 2014. The primary outcome was successful match to a CV fellowship within 1 year of completing IM residency. Independent variables included residents' licensing exam scores, research publications, medical school reputation, Alpha Omega Alpha (AOA) membership, declaration of intent to pursue CV in the residency application personal statement, clinical evaluation scores, mini-clinical evaluation exercise scores, in-training examination (ITE) performance, and exposure to CV during residency.

RESULTS

Of the 339 included residents (59% male; mean age 27) from 120 medical schools, 73 (22%) matched to CV fellowship. At the time of residency application, 104 (31%) had ≥1 publication, 38 (11%) declared intention to pursue CV in their residency application personal statement, and 104 (31%) were members of AOA. Prior to fellowship application, 111 (33%) completed a CV elective rotation. At the completion of residency training, 108 (32%) had ≥3 publications. In an adjusted logistic regression analysis, declaration of intention to pursue CV (OR 6.4, 99% CI 1.7-23.4; p < 0.001), completion of a CV elective (OR 7.3, 99% CI 2.8-19.0; p < 0.001), score on the CV portion of the PGY-2 ITE (OR 1.05, 99% CI 1.02-1.08; p < 0.001), and publication of ≥3 manuscripts (OR 4.7, 99% CI 1.1-20.5; p = 0.007) were positively associated with matching to a CV fellowship. Overall PGY-2 ITE score was negatively associated (OR 0.93, 99% CI 0.90-0.97; p < 0.001) with matching to a CV fellowship.

CONCLUSIONS

Residents' matriculation into CV fellowships was associated with declaration of CV career intent, completion of a CV elective rotation, CV medical knowledge, and research publications during residency. These findings may be useful when advising residents about pursuing careers in CV. They may also help residents understand factors associated with a successful match to a CV fellowship. The negative association between matching into CV fellowship and overall ITE score may indicate excessive subspecialty focus during IM residency.

Evaluating the recruitment process into UK anaesthesia core training: a national data linkage study of doctors' performance at selection and subsequent postgraduate training

PURPOSE OF THE STUDY

To explore which factors increase the likelihood of being deemed appointable to core anaesthesia training in the UK and whether those factors subsequently predict performance in postgraduate training.

STUDY DESIGN

Observational study linking UK medical specialty recruitment data with postgraduate educational performance, as measured by Annual Review of Competence Progression (ARCP) outcomes. Data were available for 2782 trainee doctors recruited to anaesthesia core training from 2012 to 2016 with at least one subsequent ARCP outcome.

RESULTS

Both higher interview and shortlisting scores were independent and statistically significant (p≤0.001) predictors of more satisfactory ARCP outcomes, even after controlling for the influence of postgraduate exam failure. It was noted that a number of background variables (eg, age at application) were independently associated with the odds of being deemed appointable at recruitment. Of these, increasing age and experience were also negative predictors of subsequent ARCP rating. These influences became statistically non-significant once ARCP outcomes associated with exam failure were excluded.

CONCLUSIONS

The predictors of 'appointability' largely also predict subsequent performance in postgraduate training, as indicated by ARCP ratings. This provides evidence for the validity of the selection process. Our results also suggest that greater weight could be applied to shortlisting scores within the overall process of ranking applicants for posts.

Inter-Rater Reliability of Grading Undergraduate Portfolios in Veterinary Medical Education

The reliability of high-stakes assessment of portfolios containing an aggregation of quantitative and qualitative data based on programmatic assessment is under debate, especially when multiple assessors are involved. In this study carried out at the Faculty of Veterinary Medicine, Utrecht University, the Netherlands, two independent assessors graded the portfolios of students in their second year of the 3-year clinical phase. The similarity of grades (i.e., equal grades) and the level of the grades were studied to estimate inter-rater reliability, taking into account the potential effects of the assessor's background (i.e., originating from a clinical or non-clinical department) and student's cohort group, gender, and chosen master track (Companion Animal Health, Equine Health, or Farm Animal/Public Health). Whereas the similarity between the two grades increased from 58% in the first year the grading system was introduced to around 80% afterwards, the grade level was lower over the next 3 years. The assessor's background had a minor effect on the proportion of similar grades, as well as on grading level. The assessor intraclass correlation was low (i.e., all assessors scored with a similar grading pattern [same range of grades]). The grades awarded to female students were higher but more often dissimilar. We conclude that the grading system was well implemented and has a high inter-rater reliability.