Relationship between performance on child and adolescent psychiatry in-training and certification examinations

A Systematic Review of the Relationship Between In-Training Examination Scores and Specialty Board Examination Scores

BACKGROUND

In-training examinations (ITEs) are intended for low-stakes, formative assessment of residents' knowledge, but are increasingly used for high-stake purposes, such as to predict board examination failures.

OBJECTIVE

The aim of this review was to investigate the relationship between performance on ITEs and board examination performance across medical specialties.

METHODS

A search of the literature for studies assessing the strength of the relationship between ITE and board examination performance from January 2000 to March 2019 was completed. Results were categorized based on the type of statistical analysis used to determine the relationship between ITE performance and board examination performance.

RESULTS

Of 1407 articles initially identified, 89 articles underwent full-text review, and 32 articles were included in this review. There was a moderate-strong relationship between ITE and board examination performance, and ITE scores significantly predict board examination scores for the majority of studies. Performing well on an ITE predicts a passing outcome for the board examination, but there is less evidence that performing poorly on an ITE will result in failing the associated specialty board examination.

CONCLUSIONS

There is a moderate to strong correlation between ITE performance and subsequent performance on board examinations. That the predictive value for passing the board examination is stronger than the predictive value for failing calls into question the "common wisdom" that ITE scores can be used to identify "at risk" residents. The graduate medical education community should continue to exercise caution and restraint in using ITE scores for moderate to high-stakes decisions.

Aligning In-Service Training Examinations in Plastic Surgery and Orthopaedic Surgery With Competency-Based Education

BACKGROUND

In-service training examinations (ITEs) are used to assess residents across specialties. However, it is not clear how they are integrated with the Accreditation Council for Graduate Medical Education Milestones and competencies.

OBJECTIVE

This study explored the distribution of specialty-specific milestones and competencies in ITEs for plastic surgery and orthopaedic surgery.

METHODS

In-service training examinations were publicly available for plastic surgery (PSITE) and orthopaedics (OITE). Questions on the PSITE for 2014-2016 and the OITE for 2013-2015 were mapped to the specialty-specific milestones and the 6 competencies.

RESULTS

There was an uneven distribution of milestones and competencies in ITE questions. Nine of the 36 Plastic Surgery Milestones represented 52% (341 of 650) of questions, and 3 were not included in the ITE. Of 41 Orthopaedic Surgery Milestones, 7 represented 51% (201 of 394) of questions, and 5 had no representation on the ITE. Among the competencies, patient care was the most common (PSITE = 62% [403 of 650]; OITE = 59% [233 of 394]), followed by medical knowledge (PSITE = 34% [222 of 650]; OITE = 31% [124 of 394]). Distribution of the remaining competencies differed between the 2 specialties (PSITE = 4% [25 of 650]; OITE = 9% [37 of 394]).

CONCLUSIONS

The ITEs tested slightly more than half of the milestones for the 2 specialties, and focused predominantly on patient care and medical knowledge competencies.

Are Commonly Used Premedical School or Medical School Measures Associated With Board Certification?

Purpose: To determine if there is an association between several commonly obtained premedical school and medical school measures and board certification performance. We specifically included measures from our institution for which we have predictive validity evidence into the internship year. We hypothesized that board certification would be most likely to be associated with clinical measures of performance during medical school, and with scores on standardized tests, whether before or during medical school. Methods: Achieving board certification in an American Board of Medical Specialties specialty was used as our outcome measure for a 7-year cohort of graduates (1995–2002). Age at matriculation, Medical College Admissions Test (MCAT) score, undergraduate college grade point average (GPA), undergraduate college science GPA, Uniformed Services University (USU) cumulative GPA, USU preclerkship GPA, USU clerkship year GPA, departmental competency committee evaluation, Internal Medicine (IM) clerkship clinical performance rating (points), IM total clerkship points, history of Student Promotion Committee review, and United States Medical Licensing Examination (USMLE) Step 1 score and USMLE Step 2 clinical knowledge score were associated with this outcome. Results: Ninety-three of 1,155 graduates were not certified, resulting in an average rate of board certification of 91.9% for the study cohort. Significant small correlations were found between board certification and IM clerkship points (r = 0.117), IM clerkship grade (r = 0.108), clerkship year GPA (r = 0.078), undergraduate college science GPA (r = 0.072), preclerkship GPA and medical school GPA (r = 0.068 for both), USMLE Step 1 (r = 0.066), undergraduate college total GPA (r = 0.062), and age at matriculation (r = −0.061). In comparing the two groups (board certified and not board certified cohorts), significant differences were seen for all included variables with the exception of MCAT and USMLE Step 2 clinical knowledge scores. All the variables put together could explain 4.1% of the variance of board certification by logistic regression. Conclusions: This investigation provides some additional validity evidence that measures collected for purposes of student evaluation before and during medical school are warranted.