Competency assessment in resident education

Residents' ability to interpret radiology images: development and improvement of an assessment tool

RATIONALE AND OBJECTIVES

Despite increasing radiology coverage, nonradiology residents continue to preliminarily interpret basic radiologic studies independently, yet their ability to do so accurately is not routinely assessed.

MATERIALS AND METHODS

An online test of basic radiologic image interpretation was developed through an iterative process. Educational objectives were established, then questions and images were gathered to create an assessment. The test was administered online to first-year interns (postgraduate year [PGY] 1) from 14 different specialties, as well as a sample of third- and fourth-year radiology residents (PGY3/R2 and PGY4/R3).

RESULTS

Over a 2-year period, 368 residents were assessed, including PGY1 (n = 349), PGY3/R2 (n = 14), and PGY4/R3 (n = 5) residents. Overall, the test discriminated effectively between interns (average score = 66%) and advanced residents (R2 = 86%, R3 = 89%; P < .05). Item analysis indicated discrimination indices ranging from -0.72 to 48.3 (mean = 3.12, median 0.58) for individual questions, including four questions with negative discrimination indices. After removal of the negatively indexed questions, the overall predictive value of the instrument persisted and discrimination indices increased for all but one of the remaining questions (range 0.027-70.8, mean 5.76, median 0.94).

CONCLUSIONS

Validation of an initial iteration of an assessment of basic image-interpretation skills led to revisions that improved the test. The results offer a specific test of radiologic reading skills with validation evidence for residents. More generally, results demonstrate a principled approach to test development.

Residents' perception of a novel end-of-rotation evaluation method

RATIONALE AND OBJECTIVES

With the advent of the new core and certifying examinations, a need has arisen to restructure learning and assessment to better reflect the emphasis on continuous learning throughout residency. We developed a multiparametric end-of-rotation (EOR) evaluation tool to assess medical knowledge, oral presentation, and written communication skills administered to the residents at the end of each core subspecialty rotation. The benefit of continual assessment is obvious from a program perspective; the purpose of this article is to evaluate the residents' perception of the process.

MATERIALS AND METHODS

All residents (n = 31, 28 postgraduate years two through five and 3 postgraduate year one) participate in the mandatory EOR evaluation as a required component of the residency program. After receiving Institutional Review Board approval, informed consent was obtained from the residents wishing to participate in quarterly 16-question online surveys assessing their experience. Each survey consists of 15 questions with Likert scale responses (1 through 5 from strongly disagree to strongly agree) and one free text answer. Data are collected quarterly, starting in September 2011.

RESULTS

Overall, the residents' response has been positive. The new evaluation method is felt to be more meaningful than (average 3.9, standard deviation 0.9) and is favored by the residents over the traditional competency-based evaluation (average 4.0, standard deviation 1.0). However, residents retain neutral attitudes regarding preparation for boards or changes in study habits (average score 3.6, standard deviation 0.9 and 3.6, and standard deviation 1.1, respectively).

CONCLUSION

Residents rate the EOR evaluation experience positively, although they do not report changes in study habits or increased preparedness for the new certifying examination.

Radiology residents' experience with intussusception reduction

BACKGROUND

Residents should be exposed to adequate procedural volume to act independently upon completion of training. Informal inquiry led us to question whether residents encounter enough intussusception reductions to become comfortable with the procedure.

OBJECTIVE

We sought to determine radiology residents' exposure to intussusception reductions, and whether their experiences vary by region or institution.

MATERIALS AND METHODS

U.S. radiology residency program directors were asked to encourage their residents to complete a 12-question online survey describing characteristics of their pediatric radiology department, experiences with intussusception reduction, and confidence in their own ability to perform the procedure.

RESULTS

Six hundred sixty-four residents responded during the study period. Of those, 308 (46.4%) had not experienced an intussusception reduction, and 228 (34%) had experienced only one or two. Twenty-two percent of fourth-year residents had never experienced an intussusception reduction, and 21% had experienced only one. Among second- through fourth-year residents, only 99 (18.3%) felt confident that they could competently reduce an intussusception (P < 0.0001), and 336 (62.2%) thought they would benefit from a computer-assisted training model simulating intussusception reduction (P < 0.0001).

CONCLUSION

Radiology residents have limited opportunity to learn intussusception reduction and therefore lack confidence. Most think they would benefit from additional training with a computer-simulation model.