Using an Accountability Program to Improve Psychiatry Resident Scores on In-Service Examinations

Association of Gender and Race/Ethnicity with Internal Medicine In-Training Examination Performance in Graduate Medical Education

BACKGROUND

Disparities in objective assessments in graduate medical education such as the In-Training Examination (ITE) that disadvantage women and those self-identifying with race/ethnicities underrepresented in medicine (URiM) are of concern.

OBJECTIVE

Examine ITE trends longitudinally across post-graduate year (PGY) with gender and race/ethnicity.

DESIGN

Longitudinal analysis of resident ITE metrics at 7 internal medicine residency programs, 2014-2019. ITE trends across PGY of women and URiM residents compared to non-URiM men assessed via ANOVA. Those with ITE scores associated with less than 90% probability of passing the American Board of Internal Medicine certification exam (ABIM-CE) were identified and odds of being identified as at-risk between groups were assessed with chi square.

PARTICIPANTS

A total of 689 IM residents, including 330 women and URiM residents (48%).

MAIN MEASURES

ITE score KEY RESULTS: There was a significant difference in ITE score across PGY for women and URiM residents compared to non-URiM men (F_{(2, 1321)} 4.46, p=0.011). Adjusting for program, calendar year, and baseline ITE, women and URiM residents had smaller ITE score gains (adjusted mean change in score between PGY1 and PGY3 (se), non-URiM men 13.1 (0.25) vs women and URiM residents 11.4 (0.28), p<0.001). Women and URiM residents had greater odds of being at potential risk for not passing the ABIM-CE (OR 1.75, 95% CI 1.10 to 2.78) with greatest odds in PGY3 (OR 3.13, 95% CI 1.54 to 6.37).

CONCLUSION

Differences in ITE over training were associated with resident gender and race/ethnicity. Women and URiM residents had smaller ITE score gains across PGY translating into greater odds of potentially being seen as at-risk for not passing the ABIM-CE. Differences in ITE over training may reflect differences in experiences of women and URiM residents during training and may lead to further disparities.

Exploring Pediatric Resident Attitudes and Preferences for Board Exam Preparation

Objective The American Board of Pediatrics Certifying Exam (ABP CE) is a high stakes exam and is important for employment and fellowship opportunities in pediatrics. Although research has suggested interventions which may improve scores, little research has focused on resident perception of preparation for the ABP CE. In this study, we aimed to better define pediatric residents' attitudes and preferences regarding preparation for the ABP CE. Methods Pediatric residents from one residency program were invited to participate in focus groups to discuss their attitudes and preferences on board exam preparation for the ABP CE. Focus groups were audio recorded and transcribed. Conventional content analysis was used to analyze qualitative data. From the transcripts, authors developed codes through an iterative process, which were then organized into categories. These categories were then grouped into themes. Results Nineteen residents participated in three focus groups. Focus group transcription analysis resulted in 49 codes, which were sorted into 26 categories. The categories were then grouped into four key themes: 1) the ABP CE is not immediately important early in residency; 2) more personalized guidance is preferred; 3) consistent board preparation focus from the residency program is valued; 4) learning format is important. Conclusions Residents believe preparation for the ABP CE increases in importance as they progress through residency, and they desire more personalized, consistent, and structured focus from their training program related to ABP CE preparation. Attention to these perceptions can help guide pediatric residency program leadership in developing effective board exam preparation strategies and curricula for their residents.

Measuring change in knowledge acquisition of Rwandan residents: using the American Board of Pediatrics International In-Training Examination (I-ITE) as an independent tool to monitor individual and departmental improvements during the Human Resources for Health program: an observational study

BACKGROUND

Rwanda is the only African country to use the pediatric International In-Training Examination (I-ITE). The objectives of this study were to use the scores from the I-ITE to outline the baseline level of knowledge of Rwandan residents entering the pediatric residency and the trends in knowledge acquisition from 2012 to 2018, during the Human Resources for Health (HRH) Program, an education partnership between the Rwanda Ministry of Health and a consortium of US universities.

METHODS

A retrospective descriptive analysis of the I-ITE exam scores, taken by all Rwandan pediatric residents for five of the six academic years of the study period. Individual resident scores were weighted using the non-Rwandan I-ITE sites to minimise confounding from annual variations in exam difficulty. Statistical analysis included descriptives with ANOVA to compare variation in annual mean scores.

RESULTS

Eighty-four residents took 213 I-ITE exam sittings over the five exam cycles. The mean weighted I-ITE score of all residents increased from 34% in 2013 to 49% (p < 0.001) in 2018. The 32-point gap between the mean US-ITE and Rwandan I-ITE score in 2012-2013 was reduced to a 16-point gap in 2017-2018. First year resident (PG1) scores, which likely reflect the knowledge level of undergraduate medical students entering the residency program, increased from 34.8 to 44.3% (p = 0.002) between 2013 and 2018.

CONCLUSIONS

The I-ITE is an independent, robust tool, measuring both learners and the institutional factors supporting residents. This is the first study to demonstrate that the I-ITE can be used to monitor resident knowledge acquisition in resource-limited settings, where assessment of resident knowledge can be a major challenge facing the academic medicine community. The significant increase in I-ITE scores between 2012 and 18 reflects the substantial curricular reorganisation accomplished through collaboration between Rwandan and US embedded faculty and supports the theory that programs such as HRH are highly effective at improving the quality of residency programs and undergraduate medical education.

One Small Step for Step 1

Step 1 of the United States Medical Licensing Examination (USMLE) is a multiple-choice exam primarily measuring knowledge about foundational sciences and organ systems. The test was psychometrically designed as pass/fail for licensing boards to decide whether physician candidates meet minimum standards they deem necessary to obtain the medical licensure necessary to practice. With an increasing number of applicants to review, Step 1 scores are commonly used by residency program directors to screen applicants, even though the exam was not intended for this purpose. Elsewhere in this issue, Chen and colleagues describe the "Step 1 climate" that has evolved in undergraduate medical education, affecting learning, diversity, and well-being.Addressing issues related to Step 1 is a challenge. Various stakeholders frequently spend more time demonizing one another rather than listening, addressing what lies under their respective control, and working collaboratively toward better long-term solutions. In this Invited Commentary, the author suggests how different constituencies can act now to improve this situation while aspirational future solutions are developed.One suggestion is to report Step 1 and Step 2 Clinical Knowledge scores as pass/fail and Step 2 Clinical Skills scores numerically. Any changes must be carefully implemented in a way that is mindful of the kind of unintended consequences that have befallen Step 1. The upcoming invitational conference on USMLE scoring (InCUS) will bring together representatives from all stakeholders. Until there is large-scale reform, all stakeholders should commit to taking (at least) one small step toward fixing Step 1 today.