Concurrent Validity Between a Shared Curriculum, the Internal Medicine In-Training Examination, and the American Board of Internal Medicine Certifying Examination

Resident factors associated with American board of internal medicine certification exam failure

INTRODUCTION

Performance on the certifying examinations such as the American Board of Internal Medicine Certification Exam (ABIM-CE) is of great interest to residents and their residency programs. Identification of factors associated with certification exam result may allow residency programs to recognize and intervene for residents at risk of failing. Despite this, residency programs have few evidence-based predictors of certification exam outcome. The change to pass-or-fail score reporting of the USA Medical Licensing Exam (USMLE) Step 1 removes one such predictor.

MATERIALS AND METHODS

We performed a retrospective study of residents from a medium-sized internal medicine residency program who graduated from 1998 through 2017. We used univariate tests of associations between ABIM-CE result and various demographic and scholastic factors.

RESULTS

Of 166 graduates, 14 (8.4%) failed the ABIM-CE on the first attempt. Failing the first attempt of the ABIM-CE was associated with older median age on entering residency (29 vs 27 years; P = 0.01); lower percentile rank on the Internal Medicine In-Training Examination (IM-ITE) in each of the first, second, and third years of training (P < 0.001 for all); and lower scores on the USMLE Steps 1, 2 Clinical Knowledge, and 3 (P < 0.05 for all). No association was seen between a variety of other scholastic or demographic factors and first-attempt ABIM-CE result.

DISCUSSION

Although USMLE step 1 has changed to a pass-or-fail reporting structure, there are still other characteristics that allow residency programs to identify residents at risk of ABIM-CE first time failure and who may benefit from intervention.

Does an endocrinology subspecialty residency rotation enhance resident endocrine clinical knowledge?

BACKGROUND

Internal Medicine (IM) programs offer elective subspecialty rotations in which residents may enroll to supplement the experience and knowledge obtained during general inpatient and outpatient rotations. Objective evidence that these rotations provide enhanced subspecialty specific knowledge is lacking. The purpose of this study was to determine whether exposure to an endocrinology subspecialty rotation enhanced a resident's endocrinology-specific knowledge beyond that otherwise acquired during IM residency.

METHODS

Data were collected on internal medicine resident scores on the American College of Physicians Internal Medicine In-Training Examinations (IM-ITE) for calendar years 2012 through 2018 along with enrollment data as to whether residents had completed an endocrinology subspecialty rotation prior to sitting for a given IM-ITE. Three hundred and six internal medicine residents in the University of Minnesota Internal Medicine residency program with 664 scores total on the IM-ITE for calendar years 2012 through 2018. Percentage of correct answers on the overall and endocrine subspecialty content areas on the IM-ITE for each exam were determined and the association between prior exposure to an endocrinology subspecialty rotation and percentage of correct answers in the endocrinology content area was analyzed using generalized linear mixed-effects models.

RESULTS

Two hundred and thirty-three residents (76%) completed an endocrinology subspecialty rotation at some point during their residency; 121 (40%) residents had at least one IM-ITE both before and after exposure to an endocrine subspecialty rotation. Exposure to an endocrinology subspecialty rotation exhibited a positive association with the expected IM-ITE percent correct on the endocrinology content area (5.5% predicted absolute increase). Advancing year of residency was associated with a predicted increase in overall IM-ITE score but did not improve the predictive model for endocrine subspecialty score.

CONCLUSIONS

Completion of an endocrinology subspecialty elective was associated with an increase in resident endocrine specific knowledge as assessed by the IM-ITE. These findings support the value of subspecialty rotations in enhancing a resident's subspecialty specific medical knowledge.

Low performance of internal medicine senior residents in in-service examinations

We observed an unprecedented and consistent low performance of senior residents as compared with juniors in monthly examinations. This compelled us to evaluate systematically and compare the scores of senior residents with their junior colleagues. This retrospective observational study was conducted in April 2020 among internal medicine residents. Residents in first and second year of their training were labelled as junior residents. Residents in third or fourth year of their training were labelled as senior residents. Comparison of mean scores of each resident level was done separately both for monthly formative multiple-choice questions tests, and summative yearly end of term examinations. We discussed the possible reasons as well. There were 67 residents in year 2018 and 69 in 2019. There is no significant difference between scores of monthly examinations of years 2018 and 2019 among residents of each level. Rather, in March and December 2018, junior residents perform better than senior residents with p values of 0.01 and 0.04, respectively. In February and September 2019, senior residents performed better than junior residents with p value of 0.02. Similarly, there is no significant difference in scores of residents of each level in end-of-term examinations of years 2018 and 2019 with p values 0.18 and 0.25, respectively. Performance of senior residents in our residency programme in in-service examinations is relatively low as compared with their junior colleagues. There is a need to evaluate reasons for this relatively low performance of senior residents.

Do USMLE steps, and ITE score predict the American Board of Internal Medicine Certifying Exam results?

BACKGROUND

To evaluate if United States Medical Licensing Examination (USMLE) Step 1, USMLE Step 2 CK, USMLE Step 3, and residency third-year in-service training exam (ITE) scores predict the results of American Board of Internal Medicine Certifying Exam (ABIM-CE).

METHODS

We performed a retrospective review of USMLE Step 1, USMLE Step 2 CK, USMLE Step 3, third-year residency ITE scores and ABIM-CE results of IM residents at our residency program from 2004 through 2017. Statistical analysis was perfrormed using Pearson correlation coefficient, and logistic regression to assess the relationship between USMLE Step 1, USMLE Step 2CK, USMLE Step 3, 3rd year ITE scores and ABIM-CE results. We used Multivariate logistic regression to predict pass or fail results in ABIM-CE based on USMLE and third-year ITE test scores controlling for other covariates.

RESULTS

Among 114 Internal Medicine MD residents included in the study, 92% (n = 105) passed the ABIM-CE. The OR of passing ABIM-CE was 2.70 (95% CI = 1.38-5.29), 2.31 (95% CI = 1.33-4.01), and 1.63 (95% CI = 0.81-3.29) with a ten-point increase in USMLE Step 1, USMLE Step 2 CK and USMLE Step 3 scores respectively. The OR of ABIM-CE passing chance was 2.96 (95% CI = 0.95-9.20), with a ten-point increase in the average score of the above three exams. A 5 % increase in ITE percentage raised the likelihood of passing ABIM-CE (OR 2.92, 95% CI 1.15-7.38). All residents who failed ABIM-CE had Step 1 scores < 220. Among 31 residents with Step 2 CK score < 220, 20% (n = 6) failed ABIM. Similarly, 9% of residents with USMLE Step 3 score < 220 failed ABIM-CE; all residents who failed had scored < 220. The probability curve predicted that the chance of passing ABIM- CE was around 80% with USMLE scores greater than 200 and increased to almost 100% with USMLE scores of 250 or more.

CONCLUSION

USMLE Step 1, USMLE Step 2 CK, and third-year ITE scores can predict the chances of passing ABIM-CE. The third-year ITE score has a higher preditive value compared to USMLE Step 1 and USMLE Step 2 scores. USMLE Step 1 scores more predictive of ABIM-CE results compared to USMLE Step 2CK scores. Thus, residency programs can identify internal medicine residents at risk of failing ABIM-CE and formulate interventions at an early stage during residency training. Measures such as enrolling them in question banks or board review courses can be helpful in improving their chances of passing ABIM-CE.

Exploring examinee behaviours as validity evidence for multiple-choice question examinations

CONTEXT

Clinical-vignette multiple choice question (MCQ) examinations are used widely in medical education. Standardised MCQ examinations are used by licensure and certification bodies to award credentials that are meant to assure stakeholders as to the quality of physicians. Such uses are based on the interpretation of MCQ examination performance as giving meaningful information about the quality of clinical reasoning. There are several assumptions foundational to these interpretations and uses of standardised MCQ examinations. This study explores the implicit assumption that cognitive processes elicited by clinical-vignette MCQ items are like the processes thought to occur with 'real-world' clinical reasoning as theorised by dual-process theory.

METHODS

Fourteen participants (three medical students, five residents and six staff physicians) completed three sets of five timed MCQ items (total 15) from the Medical Knowledge Self-Assessment Program (MKSAP). Upon answering a set of MCQs, each participant completed a retrospective think aloud (TA) protocol. Using constant comparative analysis (CCA) methods sensitised by dual-process theory, we performed a qualitative thematic analysis.

RESULTS

Examinee behaviours fell into three categories: clinical reasoning behaviours, test-taking behaviours and reactions to the MCQ. Consistent with dual-process theory, statements about clinical reasoning behaviours were divided into two sub-categories: analytical reasoning and non-analytical reasoning. Each of these categories included several themes.

CONCLUSIONS

Our study provides some validity evidence that test-takers' descriptions of their cognitive processes during completion of high-quality clinical-vignette MCQs align with processes expected in real-world clinical reasoning. This supports one of the assumptions important for interpretations of MCQ examination scores as meaningful measures of clinical reasoning. Our observations also suggest that MCQs elicit other cognitive processes, including certain test-taking behaviours, that seem 'inauthentic' to real-world clinical reasoning. Further research is needed to explore if similar themes arise in other contexts (e.g. simulated patient encounters) and how observed behaviours relate to performance on MCQ-based assessments.