Correlation between trainee candidate selection criteria and subsequent performance

Performance in Irish Selection and Future Performance in Surgical Training

Importance

Selection processes for surgical training should aim to identify candidates who will become competent independent practitioners and should aspire to high standards of reliability and validity.

Objective

To determine the association between measured candidate factors at the time of an Irish selection and assessment outcomes in surgical training, examined via rate of progression to Higher Specialist Training (HST), attrition rates, and performance as assessed through a multimodal framework of workplace-based and simulation-based assessments.

Design, Setting, and Participants

This retrospective observational cohort study included data from all successful applicants to the Royal College of Surgeons in Ireland (RCSI) national Core Surgical Training (CST) program. Participants included all trainees recruited to dedicated postgraduate surgical training from 2016 to 2020. These data were analyzed from July 11, 2016, through July 10, 2022.

Exposures

Selection decisions were based on a composite score that was derived from technical aptitude assessments, undergraduate academic performance, and a 4-station multiple mini-interview.

Main outcomes and measures

Assessment data, attrition rates, and rates of progression to HST were recorded for each trainee. CST performance was assessed using workplace-based and simulation-based technical and nontechnical skill assessments. Potential associations between selection and assessment measures were explored using Pearson correlation, logistic regression, and multiple linear-regression analyses.

Results

Data were available for 303 trainees. Composite scores were positively associated with progression to HST (odds ratio [OR], 1.09; 95% CI, 1.05-1.13). There was a weak positive correlation, ranging from 0.23 to 0.34, between scores and performance across all CST assessments. Multivariable linear regression analysis showed technical aptitude scores at application were associated with future operative performance assessment scores, both in the workplace (β = 0.31; 95% CI, 0.14-0.48) and simulated environments (β = 0.57; 95% CI, 0.33-0.81). There was evidence that the interpersonal skills interview station was associated with future performance in simulated communication skill assessments (β = 0.55; 95% CI, 0.22-0.87).

Conclusions and Relevance

In this study, performance at the time of Irish national selection, measured across technical and nontechnical domains in a multimodal fashion, was associated with future performance in the workplace and in simulated environments. Future studies will be required to explore the consequential validity of selection, including potential unintended effects of selection and ranking on candidate performance.

Relationship Between National Residency Matching Program (NRMP) Rank Order and Otolaryngology Residency Performance

Objective

The process of resident recruitment is costly, and our surgical residency program expends significant time on the resident selection process while balancing general duties and responsibilities. The aim of our study was to explore the relationship between otolaryngology-head and surgery (OHNS) residents' National Residency Matching Program (NRMP) rank-list position at our institution and their subsequent residency performance.

Study Design

Retrospective cohort study.

Setting

Single site institution.

Methods

We retrospectively reviewed 7 consecutive resident classes (2011-2017) at a single tertiary OHNS residency program. We reviewed each resident's absolute rank order in the NRMP matches. Measures of residency performance included overall faculty evaluation during postgraduate year 5 (PGY5), annual in-service examination scores (scaled score), and the number of manuscripts published in peer-reviewed journals. Correlations between NRMP rank order and subsequent residency performance were assessed using Spearman's rho correlation coefficients (ρ).

Results

Twenty-eight residents entered residency training between 2011 and 2017. The average rank position of the trainees during this study was 9.7 (range: 1-22). We found no significant correlation between rank order and faculty evaluation during PGY5 (ρ = 0.097, P = .625) or number of publications (ρ = -0.256, P = .189). Additionally, when assessing the association between rank order and annual Otolaryngology Training Examination-scaled scores, no statistically significant relationship was found between the 2 (P > .05).

Conclusion

Our results showed that there were no significant correlations between OHNS rank order and various measures of success in residency training, which aligns with existing literature. Further investigation of this relationship should be conducted to ensure the applicability of our findings.

Integrated cardiothoracic surgery match: Trends among applicants compared with other surgical subspecialties

OBJECTIVE

The objective of this study was to evaluate trends, qualifications, race/ethnicity, and gender of applicants to integrated cardiothoracic (CT I-6) residency programs and compare them with other competitive surgical subspecialties.

METHODS

Data were collected from the National Residency Matching Program, Electronic Residency Application Service, and Association of American Medical Colleges for thoracic surgery, orthopedic surgery, neurological surgery, otolaryngology (ENT), plastic surgery, and vascular surgery for 2010 t0 2020. Applicant gender, race/ethnicity, Alpha Omega Alpha (AOA) membership, United States Medical Licensing Examination scores, research productivity, and graduation from a top-40 medical school were analyzed.

RESULTS

From 2010 to 2020, CT I-6 experienced growth in postgraduate year 1 positions (280.0%), total applicants (62.2%), and US senior applicants (59.2%). No growth in CT I-6 positions (38) or programs (29) occurred from 2016 to 2020. CT I-6 had the lowest match rates among total applicants (31.7%) and US seniors (41.0%) in 2020. CT I-6 had fewer female applicants compared with ENT (P < .001) and plastic surgery (P < .001), but more than orthopedic surgery (P < .001). Although most CT I-6 US applicants self-identified as White (75.0%), there were more Asian applicants compared with applicants for orthopedic surgery (P < .001), ENT (P < .001), plastic surgery (P < .001), and neurological surgery (P < .01). Matched applicants averaged the highest Step 2-Clinical Knowledge scores (255.1), AOA membership (48.5%), and graduation rates from top-40 medical schools (54.5%).

CONCLUSIONS

Despite tremendous growth in positions, CT I-6 has consistently been the most difficult surgical subspecialty to match. CT I-6 has recently attracted an increasingly diverse applicant pool. For the 2019 to 2020 National Residency Matching Program Match Cycle, successful applicants had the highest Step 2-Clinical Knowledge scores, AOA membership rates, and graduation rates from a top-40 medical school among all surgical subspecialties examined.

Who will excel in advanced endoscopy? A study assessing the criteria and perceptions of experts with regard to selection of ERCP and EUS trainees

Background and study aims  Training program directors (TPDs) and experts play a crucial role in selecting ERCP/EUS trainees and determining the workforce in endoscopy. Additionally, prospective trainees should know what TPDs/experts expect from them. Nonetheless, the criteria and perceptions used in this selection have not been clarified. The aim of this study was to identify TPD/expert values/beliefs regarding personal attributes needed for selecting trainees that can excel and those which may lead to disqualification; compare perspectives between TPDs/experts and trainees regarding the selection process and critical trainee characteristics; and investigate the general approach and satisfaction regarding current application process for ERCP/EUS training.

Methods  We conducted a web-based survey to collect general opinion and data regarding the application process and trainee selection and disqualification from training. European TPDs/experts and trainees were invited to participate.

Results  Thirty-six TPDs/experts and 25 trainees from 18 countries responded. The application process is mainly driven by individual request (86.1 %). Almost half of TPDs/experts felt only moderately (38.9 %) to slightly (8.3 %) satisfied with the current application process. TPDs/experts value a diversity of trainee characteristics, but mainly traits such as “honesty,” “being a team player,” and “self-awareness” (72.2 % each). Technical skills ranked seventh as considered “extremely important.” “Disregard for patient welfare” and “lack of work ethic” were the most common reasons for disqualification. TPDs/experts and trainees agreed in most questions.

Conclusions  This survey outlines trainee selection criteria for ERCP/EUS training. Non-technical skills are the most valued by TPDs/experts. While knowledge and technical expertise are clearly important, understanding that professional attitudes are highly regarded may help direct the application process more effectively.

Ethnic/Racial Bias in Medical School Performance Evaluation of General Surgery Residency Applicants

OBJECTIVE

Differential use of communal terms (caring/unselfish traits) versus agentic terms (goal-oriented/leadership/achievement traits) may reveal bias and has been extensively reported in letters of recommendation for residency. We evaluated bias in medical student performance evaluations (MSPE) of general surgery residency applicants.

DESIGN

This is a retrospective study evaluating ethnic/race bias, as measured by differential use of agentic and communal terms, in MSPEs of residency applicants. 50% of MSPEs were randomly selected. An ethnic bias calculator derived from an open-source online gender bias calculator was populated with a list of validated agentic and communal terms. Relative frequency of communal and agentic terms was used to estimate bias. Multivariable regression was used to assess the association between the terms and ethnicity/race.

PARTICIPANTS

US medical students applying for a categorical surgery residency position at a single academic institution for a single Match cycle.

RESULTS

A total of 339 MSPEs were reviewed from 119 US medical schools. Genders were equally represented (women, 51.6%); most participants were white and Asian applicants (79.1%). Overall, MSPEs were more agency biased (65.2%) than communal biased (16.2%) or neutral (18.6%). MSPEs for Black and Hispanic/Latinx applicants were more likely to contain communal rather than agentic terms (adjusted OR: 3.02, 95% CI: 1.52-6.02) when compared to white and Asian applicants. This finding was independent of MSPE writer's gender or rank.

CONCLUSIONS

Surgery residency applicants self-identifying as Black and Hispanic/Latinx were more likely to be described using communal traits compared to white and Asian applicants, suggesting ethnic/racial bias. Such differences in language utilized in MSPEs may impact residency opportunities for applicants who are under-represented in medicine. Educational efforts aimed at MSPE writers may help to reduce bias.

Disrupting the Fellowship Match: COVID-19 and the Applicant Arms Race

Resident and fellow selection carried out via "the Match" has historically relied upon in-person interviews to evaluate and rank candidates. However, the COVID-19 pandemic has required fellowship matches and the upcoming Main Residency Match® to become wholly virtual. The 2020 pediatric surgery match offers a unique case study in the benefits and shortcomings of a virtual process and begins a much-needed conversation regarding opportunities for innovation in candidate selection. For many candidates, the application cycle imposes considerable costs - financial, professional, and personal - which have only escalated over time. We draw on our experience from the most recent match cycle to discuss limitations of the traditional Match® and suggest potential solutions to improve the subspecialty interview process moving forward.

Different approaches to selection of surgical trainees in the European Union

BACKGROUND

There is an increasing global interest in selection processes for candidates to surgical training. The aim of the present study is to compare selection processes to specialist surgeon training in the European Union (EU). A secondary goal is to provide guidance for evidence-based methods by a proposed minimum standard that would align countries within the EU.

METHODS

Publications and grey literature describing selection strategies were sought. Correspondence with Union Européenne des Médecins Specialists (UEMS) Section of Surgery delegates was undertaken to solicit current information on national selection processes. Content analysis of 13 semi-structured interviews with experienced Swedish surgeons on the selection process. Two field trips to Ireland, a country with a centralized selection process were conducted. Based on collated information typical cases of selection in a centralized and decentralized setting, Ireland and Sweden, are described and compared.

RESULTS

A multitude of methods for selection to surgical training programs were documented in the 27 investigated countries, ranging from locally run processes with unstructured interviews to national systems for selection of trainees with elaborate structured interviews, and non-technical and technical skills assessments. Associated with the difference between centralized and decentralized selection systems is whether surgical training is primarily governed by an employment or educational logic. Ireland had the most centralized and elaborate system, conducting a double selection process using evidence-based methods along an educational logic. On the opposite end of the scale Sweden has a decentralized, local selection process with a paucity of evidence-based methods, no national guidelines and operates along an employment logic, and Spain that rely solely on examination tests to rank candidates.

CONCLUSION

The studied European countries all have different processes for selection of surgical trainees and the use of evidence-based methods for selection is variable despite similar educational systems. Selection in decentralized systems is currently often conducted non-transparent and subjectively. A suggested improvement towards an evidence-based framework for selection applicable in centralized and decentralized systems as well as educational and employer logics is suggested.

Predicting Quality of Clinical Performance From Cardiology Fellowship Applications

Variables in cardiology fellowship applications have not been objectively analyzed against applicants' subsequent clinical performance. We investigated possible correlations in a retrospective cohort study of 65 cardiology fellows at the Mayo Clinic (Rochester, Minn) who began 2 years of clinical training from July 2007 through July 2013. Application variables included the strength of comparative statements in recommendation letters and the authors' academic ranks, membership status in the Alpha Omega Alpha Honor Medical Society, awards earned, volunteer activities, United States Medical Licensing Examination (USMLE) scores, advanced degrees, publications, and completion of a residency program ranked in the top 6 in the United States. The outcome was clinical performance as measured by a mean of faculty evaluation scores during clinical training. The overall mean evaluation score was 4.07 ± 0.18 (scale, 1-5). After multivariable analysis, evaluation scores were associated with Alpha Omega Alpha designation (β=0.13; 95% CI, 0.01-0.25; P=0.03), residency program reputation (β=0.13; 95% CI, 0.05-0.21; P=0.004), and strength of comparative statements in recommendation letters (β=0.08; 95% CI, 0.01-0.15; P=0.02), particularly in letters from residency program directors (β=0.05; 95% CI, 0.01-0.08; P=0.009). Objective factors to consider in the cardiology fellowship application include Alpha Omega Alpha membership, residency program reputation, and comparative statements from residency program directors.

Evaluation of Validity Evidence for Personality, Emotional Intelligence, and Situational Judgment Tests to Identify Successful Residents

Importance

The ability to identify candidates who will thrive and successfully complete their residency is especially critical for general surgery programs.

Objective

To assess the extent to which 3 screening tools used extensively in industrial selection settings-emotional intelligence (EQ), personality profiles, and situational judgment tests (SJTs)-could identify successful surgery residents.

Design, Setting, and Participants

In this analysis, personality profiles, EQ assessments, and SJTs were administered from July through August 2015 to 51 postgraduate year 1 through 5 general surgery residents in a large general surgery residency program. Associations between these variables and residency performance were investigated through correlation and hierarchical regression analyses.

Interventions

Completion of EQ, personality profiles, and SJT assessments.

Main Outcomes and Measures

Performance in residency as measured by a comprehensive performance metric. A score of zero represented a resident whose performance was consistent with that of their respective cohort's performance; below zero, worse performance; and greater than zero, better performance.

Results

Of the 61 eligible residents, 51 (84%) chose to participate and 22 (43%) were women. US Medical Licensing Examination Step 1 (USMLE1), but not USMLE2, emerged as a significant factor (t2,49 = 1.98; β = 0.30; P = .03) associated with overall performance. Neither EQ facets nor overall EQ offered significant incremental validity over USMLE1 scores. Inclusion of the personality factors did not significantly alter the test statistic and did not explain any additional portion of the variance. By contrast, inclusion of SJT scores accounted for 15% more of the variance than USMLE1 scores alone, resulting in a total of 25% of the variance explained by both USMLE1 and SJT scores (F2,57 = 7.47; P = .002). Both USMLE1 (t = 2.21; P = .03) and SJT scores (t = 2.97; P = .005) were significantly associated with overall resident performance.

Conclusions and Relevance

This study found little support for the use of EQ assessment and only weak support for some distinct personality factors (ie, agreeableness, extraversion, and independence) in surgery resident selection. Performance on the SJT was associated with overall resident performance more than traditional cognitive measures (ie, USMLE scores). These data support further exploration of these 2 screening assessments on a larger scale across specialties and institutions.

Evaluating the recruitment process into UK anaesthesia core training: a national data linkage study of doctors' performance at selection and subsequent postgraduate training

PURPOSE OF THE STUDY

To explore which factors increase the likelihood of being deemed appointable to core anaesthesia training in the UK and whether those factors subsequently predict performance in postgraduate training.

STUDY DESIGN

Observational study linking UK medical specialty recruitment data with postgraduate educational performance, as measured by Annual Review of Competence Progression (ARCP) outcomes. Data were available for 2782 trainee doctors recruited to anaesthesia core training from 2012 to 2016 with at least one subsequent ARCP outcome.

RESULTS

Both higher interview and shortlisting scores were independent and statistically significant (p≤0.001) predictors of more satisfactory ARCP outcomes, even after controlling for the influence of postgraduate exam failure. It was noted that a number of background variables (eg, age at application) were independently associated with the odds of being deemed appointable at recruitment. Of these, increasing age and experience were also negative predictors of subsequent ARCP rating. These influences became statistically non-significant once ARCP outcomes associated with exam failure were excluded.

CONCLUSIONS

The predictors of 'appointability' largely also predict subsequent performance in postgraduate training, as indicated by ARCP ratings. This provides evidence for the validity of the selection process. Our results also suggest that greater weight could be applied to shortlisting scores within the overall process of ranking applicants for posts.

Utility of selection methods for specialist medical training: A BEME (best evidence medical education) systematic review: BEME guide no. 45

BACKGROUND

Selection into specialty training is a high-stakes and resource-intensive process. While substantial literature exists on selection into medical schools, and there are individual studies in postgraduate settings, there seems to be paucity of evidence concerning selection systems and the utility of selection tools in postgraduate training environments.

AIM

To explore, analyze and synthesize the evidence related to selection into postgraduate medical specialty training.

METHOD

Core bibliographic databases including PubMed; Ovid Medline; Embase, CINAHL; ERIC and PsycINFO were searched, and a total of 2640 abstracts were retrieved. After removing duplicates and screening against the inclusion criteria, 202 full papers were coded, of which 116 were included.

RESULTS

Gaps in underlying selection frameworks were illuminated. Frameworks defined by locally derived selection criteria, and heavily weighed on academic parameters seem to be giving way to the evidencing of competency-based selection approaches in some settings. Regarding selection tools, we found favorable psychometric evidence for multiple mini-interviews, situational judgment tests and clinical problem-solving tests, although the bulk of evidence was mostly limited to the United Kingdom. The evidence around the robustness of curriculum vitae, letters of recommendation and personal statements was equivocal. The findings on the predictors of past performance were limited to academic criteria with paucity of long-term evaluations. The evidence around nonacademic criteria was inadequate to make an informed judgment.

CONCLUSIONS

While much has been gained in understanding the utility of individual selection methods, though the evidence around many of them is equivocal, the underlying theoretical and conceptual frameworks for designing holistic and equitable selection systems are yet to be developed.

The Perception and Costs of the Interview Process for Plastic Surgery Residency Programs: Can the Process Be Streamlined?

BACKGROUND

The purpose of this study was to assess applicant perceptions and costs associated with the interview process for plastic surgery residency positions.

METHODS

This was a cross-sectional survey of applicants to the integrated- and independent-track residencies at the authors' institution. All applicants who were interviewed were invited to complete a Web-based survey on costs and perceptions of various components of the interview process. Descriptive and bivariate statistics were computed to compare applicants to the two program tracks.

RESULTS

Fifty-three applicants were interviewed for residency positions; 48 completed a survey (90.5 percent response rate). Thirty-four applicants were candidates for the integrated program; 16 applicants were candidates for the independent program. The program spent $2763 per applicant interviewed; 63 percent of applicants spent more than $5000 on the interview process. More than 70 percent of applicants missed more than 7 days of work to attend interviews. Independent applicants felt less strongly that interviews were critical to the selection process and placed less value on physically visiting the hospital and direct, in-person interaction. Applicants placed little value on program informational talks. Applicants who had experience with virtual interviews felt more positively about the format of a video interview relative to those who did not.

CONCLUSIONS

The residency interview process is resource intensive for programs and applicants. Removing informational talks may improve the process. Making physical tours and in-person interviews optional are other alternatives that merit future study.

Selection for Surgical Training: An Evidence-Based Review

PURPOSE

The predictive relationship between candidate selection criteria for surgical training programs and future performance during and at the completion of training has been investigated for several surgical specialties, however there is no interspecialty agreement regarding which selection criteria should be used. Better understanding the predictive reliability between factors at selection and future performance may help to optimize the process and lead to greater standardization of the surgical selection process.

METHODS

PubMed and Ovid MEDLINE databases were searched. Over 560 potentially relevant publications were identified using the search strategy and screened using the Cochrane Collaboration Data Extraction and Assessment Template.

RESULTS

57 studies met the inclusion criteria. Several selection criteria used in the traditional selection demonstrated inconsistent correlation with subsequent performance during and at the end of surgical training. The following selection criteria, however, demonstrated good predictive relationships with subsequent resident performance: USMLE examination scores, Letters of Recommendation (LOR) including the Medical Student Performance Evaluation (MSPE), academic performance during clinical clerkships, the interview process, displaying excellence in extracurricular activities, and the use of unadjusted rank lists.

CONCLUSIONS

This systematic review supports that the current selection process needs to be further evaluated and improved. Multicenter studies using standardized outcome measures of success are now required to improve the reliability of the selection process to select the best trainees.

Selection of subspecialty fellows in anesthesia for pediatric anesthesia: a national survey of program directors in the United States

BACKGROUND

Several studies have investigated the selection process for medical students, residents, and fellows across different specialties. Currently, the selection criteria used by pediatric anesthesia program directors to select prospective pediatric anesthesia fellows are unknown.

OBJECTIVE

To evaluate the criteria used by program directors in the selection of prospective pediatric anesthesia fellows in the United States.

METHODS

Survey responses were solicited from 51 pediatric anesthesia fellowship directors in the United States. Questions evaluated department demographic information, the importance of fellowship selection criteria, and program satisfaction with the Match process.

RESULTS

Forty-two of 51 fellowship directors responded to the survey (a response rate of 82%). The personal statement was reported as extremely/very important for 30 of 42 (71%) program directors, while research experience and prior publications were reported as extremely/very important for 16 of 41 (39%) and 16 of 41 (39%) program directors, respectively. Programs that offered all the spots through the Match were smaller, median (IQR) number of fellows of 3 (2-6) compared to the ones that did not offer all the spots through the Match, 8 (2-12), P = 0.03. In contrast, no difference in program directors' satisfaction with the Match was detected between smaller (≤ 3 fellows) and larger programs (>3 fellows), P = 0.25.

CONCLUSION

A minority of pediatric fellowship program directors rated prior research experience as an important factor in the selection of prospective fellows. The lack of emphasis on research qualities for fellows is potentially concerning for the future of our specialty.