Predictors of Success on the ABR Core Examination

Beyond milestones and percentiles: Revisiting non-cognitive and non-interpretive skills in radiology

While there is no precise formula for a great radiology resident, certain attributes and achievements may herald success during training. We briefly review prior works exploring predictive factors and evaluation metrics of top resident performance, noting that those focusing on non-cognitive attributes are over twenty years old. As radiology practice and education has substantially evolved in the interim, we revisit this topic from a contemporary perspective. Inspired by the literature and our own personal experiences, we suggest that the following non-cognitive traits are invaluable for radiology trainees: communication expertise, workplace adaptability, self-awareness, tech savvy and genuine interest in one's individual work and greater community. These characteristics should be highlighted by applicants, sought by selection committees, cultivated by mentors, evaluated by programs and valued by colleagues.

Performance of GPT-4 on the American College of Radiology In-training Examination: Evaluating Accuracy, Model Drift, and Fine-tuning

RATIONALE AND OBJECTIVES

In our study, we evaluate GPT-4's performance on the American College of Radiology (ACR) 2022 Diagnostic Radiology In-Training Examination (DXIT). We perform multiple experiments across time points to assess for model drift, as well as after fine-tuning to assess for differences in accuracy.

MATERIALS AND METHODS

Questions were sequentially input into GPT-4 with a standardized prompt. Each answer was recorded and overall accuracy was calculated, as was logic-adjusted accuracy, and accuracy on image-based questions. This experiment was repeated several months later to assess for model drift, then again after the performance of fine-tuning to assess for changes in GPT's performance.

RESULTS

GPT-4 achieved 58.5% overall accuracy, lower than the PGY-3 average (61.9%) but higher than the PGY-2 average (52.8%). Adjusted accuracy was 52.8%. GPT-4 showed significantly higher (p = 0.012) confidence for correct answers (87.1%) compared to incorrect (84.0%). Performance on image-based questions was significantly poorer (p < 0.001) at 45.4% compared to text-only questions (80.0%), with adjusted accuracy for image-based questions of 36.4%. When the questions were repeated, GPT-4 chose a different answer 25.5% of the time and there was no change in accuracy. Fine-tuning did not improve accuracy.

CONCLUSION

GPT-4 performed between PGY-2 and PGY-3 levels on the 2022 DXIT, significantly poorer on image-based questions, and with large variability in answer choices across time points. Exploratory experiments in fine-tuning did not improve performance. This study underscores the potential and risks of using minimally-prompted general AI models in interpreting radiologic images as a diagnostic tool. Implementers of general AI radiology systems should exercise caution given the possibility of spurious yet confident responses.

Perspectives From the RadDiscord Annual Survey: Overview of the Top Study Tools and Evaluation of Study Time and Various Resources

RATIONALE AND OBJECTIVES

Each year, senior radiology residents take the American Board of Radiology Qualifying (Core) exam to evaluate competency. Approximately 10% of first-time examinees will fail this exam (1). Understanding factors that contribute to success will help residency program directors and trainees prepare for future exams. RadDiscord (www.raddiscord.org), an international radiology educational community, is in the unique position to evaluate different study materials and resources. The goal of this paper is to report the results from the RadDiscord survey and analyze the factors that correlate with higher exam performance and passing.

MATERIALS AND METHODS

Following the February 2021, June 2021, and June 2022 exams, RadDiscord members were provided an anonymous survey, collecting information on study resources and exam scores. The collected data were analyzed using various statistical methods. Both descriptive and inferential analyses were performed.

RESULTS

A total of 318 residents responded (95% passed). Significant variability in Qualifying (Core) exam performance and perceived quality of internal didactics existed between program types. Residents who did less than 2000 practice questions performed lower on the exam. The Diagnostic Radiology In-Training (DXIT) exam was the most predictive for passing and performance. Qualifying (Core) exam performance negatively correlated with study time, though certain residents did receive some benefit from study time.

CONCLUSION

Many factors correlate with passing and Qualifying (Core) exam performance. Residency programs with fewer resources should consider alternative ways to support residents beyond offering study time. Residents who complete at least 2000 practice questions are more likely to pass and DXIT results can be a useful gauge to identify exam readiness.

Radiology program director's perspective on a novel hands-on advanced elective

With the advent of the USMLE Step 1 exam moving to a pass/fail status, Radiology Program Directors (PDs) and Associate Program Directors (APDs) need alternative methods of identifying interested and engaged medical students who are applying to their program. Additionally, undergraduate radiology medical education in the United States varies widely from institution to institution with no universal mandatory radiology component. To address these problems, we implemented an advanced fourth year hands-on radiology elective where the students were treated as first year radiology residents (R1s), giving them resident-level access to the Picture Archive and Communication System (PACS) and dictation software, and allowing them to perform entry-level procedures with appropriate supervision. After implementation of the elective, a 5-question online survey was sent to two hundred and ninety-eight PDs and APDs via the Association of Program Directors in Radiology (APDR) listserv, of which seventy-two responses were compiled, yielding a response rate of 24%. The survey focused on how a hands-on medical student elective would help in assessing prospective candidates and predicting R1 performance. Most respondents felt interest in radiology, motivation, and interpersonal skills would be better assessed after such an elective and the vast majority felt hands-on Advanced Elective would be at least slightly predictive of first year resident performance. Based on this information, we believe implementing a hands-on advanced radiology elective would significantly help address the passive nature of traditional radiology electives, providing valuable information to PDs and APDs and giving the best possible radiology experience to our medical students.

The Association Between USMLE Step 2 Clinical Knowledge Scores and Residency Performance: A Systematic Review and Meta-Analysis

PURPOSE

With the change in Step 1 score reporting, Step 2 Clinical Knowledge (CK) may become a pivotal factor in resident selection. This systematic review and meta-analysis seeks to synthesize existing observational studies that assess the relationship between Step 2 CK scores and measures of resident performance.

METHOD

The authors searched MEDLINE, Web of Science, and Scopus databases using terms related to Step 2 CK in 2021. Two researchers identified studies investigating the association between Step 2 CK and measures of resident performance and included studies if they contained a bivariate analysis examining Step 2 CK scores' association with an outcome of interest: in-training examination (ITE) scores, board certification examination scores, select Accreditation Council for Graduate Medical Education core competency assessments, overall resident performance evaluations, or other subjective measures of performance. For outcomes that were investigated by 3 or more studies, pooled effect sizes were estimated with random-effects models.

RESULTS

Among 1,355 potential studies, 68 met inclusion criteria and 43 were able to be pooled. There was a moderate positive correlation between Step 2 CK and ITE scores (0.52, 95% CI 0.45-0.59, P < .01). There was a moderate positive correlation between Step 2 CK and ITE scores for both nonsurgical (0.59, 95% CI 0.51-0.66, P < .01) and surgical specialties (0.41, 95% CI 0.33-0.48, P < .01). There was a very weak positive correlation between Step 2 CK scores and subjective measures of resident performance (0.19, 95% CI 0.13-0.25, P < .01).

CONCLUSIONS

This study found Step 2 CK scores have a statistically significant moderate positive association with future examination scores and a statistically significant weak positive correlation with subjective measures of resident performance. These findings are increasingly relevant as Step 2 CK scores will likely become more important in resident selection.

Attending Reviews of Residents Correlate with ABR Qualifying (Core) Examination Failure

RATIONALE AND OBJECTIVES

Since the American Board of Radiology (ABR) instituted the new system of board certification, there has been much discussion as to the test's validity. We decided to evaluate if subjective evaluation of resident performance correlated with ABR Qualifying (Core) Examination performance at this single institution.

MATERIALS AND METHODS

Data regarding resident evaluation scores by attending physicians and passage of board examinations was gathered regarding residents who had taken the ABR Qualifying (Core) Examination from 2013 through 2019 for a total of 42 residents, eight of whom failed the ABR Qualifying (Core) Examination on their first attempt. A univariate analysis comparing scores with resident passage or failure of the ABR Qualifying (Core) Examination on the first attempt and analyses correcting for class year only and class year and number of evaluations was performed.

RESULTS

The non-weighted average evaluation score of years 1, 2, and 3 was 80.24% for those who failed the ABR Qualifying (Core) Examination and 83.71 % for those who passed. On univariate analysis along with analyses correcting for class year only and class year along with number of evaluations, there was a statistically significant correlation with decreased evaluation scores averaged over the three years of residency and failure of the ABR Qualifying (Core) Examination (p = 0.0102, p = 0.003, and p = 0.0043). The statistical significance held for the average numerical score in each individual year of training in all analyses except for year 1 of the univariate analysis (p = 0.1264).

CONCLUSION

At the studied institution, there was a statistically significant correlation between lower subjective faculty evaluation scores and failure of the ABR Qualifying (Core) Examination.

Potential Impact of a Pass or Fail United States Medical Licensing Exam Step 1 Scoring System on Radiology Residency Applications

RATIONALE AND OBJECTIVES

To assess the anticipated impact of the change in United States Medical Licensing Examination Step 1 scoring from numerical to pass or fail on the future selection of radiology residents.

MATERIALS AND METHODS

An anonymous electronic 14-item survey was distributed to 308 members of the Association of Program Directors in Radiology and included questions regarding the anticipated importance of various application metrics when Step 1 becomes pass or fail. Secondary analyses compared responses based on the current use of a Step 1 scoring screen.

RESULTS

Eighty eight respondents (28.6% [88/308]) completed the survey. Most (64% [56/88]) noted that the United States Medical Licensing Examination Step 2 Clinical Knowledge (CK) score will likely be one of the top three most important factors in assessing applicants, followed by class ranking or quartile (51% [45/88]) and the Medical Student Performance Evaluation/Dean's Letter (42% [37/88]). Over 90% (82/88) of respondents anticipate potentially or definitively requiring Step 2 CK scores before application review, and 50% (44/88) of respondents anticipate extending interview invites at a later date to receive Step 2 CK scores. These results did not significantly differ between programs who currently use a Step 1 scoring screen and those who do not.

CONCLUSION

As Step 1 transitions from a numerical score to pass or fail, radiology residency program directors will likely rely on Step 2 CK scores as an objective and standardized metric to screen applicants. Further investigation is needed to identify other objective metrics to evaluate applicants before Step 1 becomes pass or fail.

Prospective investigation of factors associated with success on the American College of Veterinary Emergency and Critical Care certification examination (2016-2018)

OBJECTIVE

To assess the association of candidate attributes and residency training factors with success on the American College of Veterinary Emergency and Critical Care (ACVECC) board certification examination and to develop multivariable models of first-attempt success.

DESIGN

Prospective survey-based study.

SETTING

Post-assessment ACVECC examination candidates.

ANIMALS

None.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

Comprehensive surveys were distributed to ACVECC examination candidates in 2016 to 2018 after completion of their assessments, but prior to publication of examination results. Unique anonymous candidate identification numbers were used to match survey responses to outcome data from the office of the ACVECC Executive Secretary. After curation to retain only the first response from each candidate, there were 97 unique candidate responses available for analysis. Univariate analyses identified multiple factors associated with first-attempt success and multiple differences between academic and private practice residency programs. Multivariable logistic regression modeling suggested that 5 factors were independently associated with first-attempt success on the ACVECC examination, specifically younger age, more weeks of study prior to the examination, training at a facility with more ACVECC Diplomates, training at a facility with more ACVECC residents, and having no requirement to manage both Emergency Room (ER) and Critical Care (CC) cases simultaneously.

CONCLUSIONS

Numerous resident and training center factors are associated with success in the ACVECC board certification examination. Residents and training centers might be able to use these data to enhance training, but caution must be exercised because these data are associative only.

Results of the 2019 Survey of the American Alliance of Academic Chief Residents in Radiology

RATIONALE AND OBJECTIVES

An annual survey of chief residents in accredited North American radiology programs is conducted by the American Alliance of Academic Chief Residents in Radiology (A³CR²). The purpose of this study is to summarize the 2019 A³CR² chief resident survey.

MATERIALS AND METHODS

An online survey was distributed to chief residents from 194 Accreditation Council on Graduate Medical Education-accredited radiology residencies. Questions were designed to gather information about residency program details, call and weekend coverage, interventional radiology training, fellowship, social media use, healthcare reform, artificial intelligence, and job market status.

RESULTS

One hundred and forty-two unique responses from 99 programs were provided, yielding a 51% program response rate. There was a mean of 7.3 women per residency with a mean program size of 28 residents (26% women). Only 3 of the 99 (3%) programs had a proportion of women that was 50% or higher. The proportion of women in radiology residencies is unchanged since 2014 (p= 0.93) and is significantly lower than 2019 graduating women medical students (49.3%; p < 0.001). Thirty-five percent of programs had 24/7 attending coverage and 40% of programs had extended hours attending shifts. Of programs without 24/7 attending coverage, the proportion of programs without face-to-face readout has increased from 34% in 2014 to 55% in 2019 (p = 0.015). The majority (67%) of respondents had no concerns about the radiology job market; compared to 2014, where only 4% had no concerns (p < 0.001).

CONCLUSION

Women remain underrepresented in radiology, face-to-face readout is decreasing, and there has been a shift towards a positive job market outlook.

The Value of Numerical USMLE Step 1 Scores in Radiology Resident Selection

RATIONALE AND OBJECTIVES

In response to a recent proposal to change scoring on the United States Medical Licensing (USMLE) Step 1 exam to pass/fail, we sought to determine the value of numerical Step 1 scores in predicting success in our radiology residency program.

MATERIALS AND METHODS

Residency applications for 157 residents entering the program between 2005 and 2017 were retrospectively reviewed. Biographical (gender, sports participation, advanced degree), undergraduate (school, major), and medical school (grades, USMLE Step 1 score, Alpha Omega Alpha membership, letters of recommendation, publications) data were recorded. Multivariate regression analysis was used to examine the relationship between these application factors and subsequent performance as a radiology resident, as determined by completion of the program without requiring corrective action, select Accreditation Council for Graduate Medical Education milestones, and selection as chief resident.

RESULTS

Corrective action was required for 7% (n = 12) of residents. Of the predictor variables, only Step 1 score was associated with the need for corrective action (p < 0.001). The interpretation of exams milestone was associated with higher Step 1 scores (p = 0.001) and number of medical school clerkship honors (p = 0.008). Selection as chief resident was associated with sports participation (p = 0.04), and clerkship honors (p = 0.02).

CONCLUSION

Numerical USMLE Step 1 scores are predictive of successful completion of radiology residency training without the need for corrective action, and of accelerated competence in the interpretation of exams milestone. Continued reporting of numerical Step 1 scores would be valuable in selection of radiology residents.

The Relationship Between US Medical Licensing Examination Step Scores and ABR Core Examination Outcome and Performance: A Multi-institutional Study

PURPOSE

We analyzed multi-institutional data to understand the relationship of US Medical Licensing Examination (USMLE) Step scores to ABR Core examination performance to identify Step score tiers that stratify radiology residents into different Core performance groups.

METHODS

We collected USMLE Step scores and ABR Core examination outcomes and scores for anonymized residents from 13 different diagnostic radiology residency programs taking the ABR Core examination between 2013 and 2019. USMLE scores were grouped into noniles using z scores and then aggregated into three tiers based on similar Core examination pass-or-fail outcomes. Core performance was grouped using standard deviation from the mean and then measured by the percent of residents with scores below the mean. Differences between Step tiers for Core outcome and Core performance were statistically evaluated (P < .05 considered significant).

RESULTS

Differences in Step 1 terciles Core failure rates (45.9%, 11.9%, and 3.0%, from lowest to highest Step tiers; n = 416) and below-mean Core performance (83.8%, 54.1%, and 21.1%, respectively; n = 402) were significant. Differences in Step 2 groups Core failure rates (30.0%, 10.6%, and 2.0%, from lowest to highest Step tiers; n = 387) and below-mean Core performance (80.0%, 43.7%, and 14.0%, respectively; n = 380) were significant. Step 2 results modified Core outcome and performance predictions for residents in Step 1 terciles of varying statistical significance.

CONCLUSIONS

Tiered scoring of USMLE Step results has value in predicting radiology resident performance on the ABR Core examination; effective stratification of radiology resident applicants can be done without reporting numerical Step scores.

The USMLE Step 1 Pass/Fail Reporting Proposal: Another View

The Association of Program Directors in Radiology recently issued a statement endorsing continued reporting of results of the United States Medical Licensing Examination (USMLE) as a three-digit score. While this position was approved by the Association of Program Directors in Radiology Board of Directors, it does not reflect the opinions of all radiology program directors. Here, we present an argument in support of reporting USMLE results as pass/fail. As a psychometric instrument, the USMLE Step 1 is designed to assess basic science knowledge and intended to inform a binary decision on licensure. Due to a steadily-increasing burden of applications to review, program directors have increasingly relied upon scores for candidate screening. Such use has multiple adverse consequences. Student focus on Step 1 systematically devalues educational content not evaluated on the exam, and the reliance on Step 1 scores almost certainly works against efforts to increase workforce diversity. Moreover, the increasing pressure of "Step 1 Mania" has negative consequences for trainee mental health and wellness. Despite the widespread use of Step 1 scores to select applicants, there are little data to correlate scores to meaningful outcomes related to patient care or clinical practice. We find the current situation untenable, and believe a necessary first step toward reform is making Step 1 a pass/fail only examination.