The Coordinators of Psychiatric Education (COPE) Residency In-Training Exam: a preliminary psychometric assessment

Evaluating the utility, reliability, and validity of a resident performance evaluation instrument

OBJECTIVE

The primary goal of this project was to create and evaluate the utility, validity, and reliability of an instrument assessing resident performance during child and adolescent psychiatry (CAP) training. In three stages, the instrument was developed and evaluated for utility in assessing skills critical to CAP. Items on the Global Assessment Evaluation (GAE) were derived from the six core competencies identified by the American College of Graduate Medical Education (ACGME). Secondary goals included producing an instrument that could be used by other CAP training programs and describing a process that could be followed by training programs to create their own assessment instruments.

METHODS

Faculty members developed a CAP resident performance evaluation instrument. In a three-stage process, faculty utilized the tool to evaluate residents during and after rotations. Statistical findings guided revisions to improve the utility and reliability of the instrument. For the final version of the GAE, intra-class correlation coefficients were calculated to assess inter-rater reliability, and Principal Components Analysis provided further insight into the dimensions of resident assessment.

RESULTS

The final version of the GAE showed overall and construct validity by capturing significant differences among residents and matched faculty members' overall impressions of resident performance. Intra-class correlation coefficient values for the overall score (0.945) and individual scales showed good reliability. Resident performance was not correlated with rotation site or model of care.

CONCLUSIONS

The GAE has proved a valuable instrument in tracking the progress, strengths, and weaknesses in resident performance over the course of training. Data from multiple evaluations over time provide useful information about resident performance in a way that one or few evaluations does not. This finding is consistent with the practice of semiannual reviews and the new Clinical Competency Committees, both of which are required by the ACGME.

Construct validation of progress testing to measure knowledge and visual skills in radiology

BACKGROUND

The Dutch Radiology Progress Test (DRPT) monitors the acquisition of knowledge and visual skills of radiology residents in the Netherlands.

AIM

We aimed to evaluate the quality of progress testing in postgraduate radiology training by studying the reliability of the DRPT and finding an indication for its construct validity. We expected that knowledge would increase rapidly in the first years of residency, leveling-off in later years, to allow for the development of visual skills. We hypothesized that scores on the DRPT reflect this pattern.

METHODS

Internal consistencies were estimated with Cronbach's alpha. Performance increase over program years were tested using one-way analysis of variance.

RESULTS

Data were available for 498 residents (2281 test results). Reliabilities were around Cronbach's alpha 0.90. There was a significant difference in the mean test results between the first three years of residency. After the fourth year no significant increase in test scores on knowledge could be measured on eight tests. The same pattern occurred for scores on visual skills. However, visual skills scores tend to increase more sharply than knowledge scores.

CONCLUSION

We found support for the reliability and construct validity of the DRPT. However, assessment on visual skill development needs further exploration.