Evaluation of an online faculty appraisal instrument: comparison of resident and faculty perceptions

A mixed-methods analysis of residents' written comments regarding their clinical supervisors

BACKGROUND

Clinical faculty require feedback to guide their development. Although written comments submitted by residents on clinical teaching assessments (CTAs) are potentially a rich source of feedback, little is known about their information quality.

METHOD

Naturalistic study involving thematic content analyses and concordancing of comments submitted in a sample of 1,601 CTAs.

RESULTS

About half of the CTAs contained comments, and most were related to perceived teacher strengths. Mixed-methods analysis of comments in the "areas for improvement" field revealed that rather than offering constructive criticism, residents vocalized their perceived learning needs. Specific, behaviorally based comments were uncommon.

CONCLUSIONS

In their present format, the written comments analyzed by this study seem unlikely to provide faculty with substantive feedback. Greater insight into residents' understanding of the CTA process, including motivational factors, is necessary if academic centers intend to maximize the formative value of CTAs.

Impact of Electronic Resident Evaluation Systems on the Timeliness, Rate, and Quality of Responses

OBJECTIVE

To determine the impact of electronic resident evaluation systems on the timeliness, rate, and quality of responses.

METHODS

Surveys were mailed electronically to the membership of the Association of Program Directors in Radiology, which included the directors of 158 residency programs. Respondents were instructed to send 1 response from each program. Information gathered included the use of electronic compared with paper-based evaluation systems and the overall level of satisfaction with such systems (rated on a 5-point scale with 1 = dissatisfied and 5 = satisfied). Questions were asked regarding the impact of electronic systems on the timeliness, rate, and quality of responses.

RESULTS

Seventy-seven responses were received, for an estimated response rate of 49% on a per-program basis. Of these 77 respondents, 45 (58%) used electronic systems and 32 (42%) used paper-based systems. The median level of satisfaction was equivalent (4 = somewhat satisfied) for both groups. Of the 45 respondents using electronic systems, 26 (58%) reported increased response rates, 16 (36%) reported no change, and 3 (7%) reported decreased response rates compared with paper-based systems. Regarding the timeliness of responses, 31 (69%) reported faster response rates, 10 (22%) reported no change, and 4 (9%) reported slower rates compared with paper-based systems. Regarding the quality of responses, 25 (56%) reported no change, 12 (27%) reported improved quality, and 8 (18%) reported worse quality compared with paper-based systems.

CONCLUSION

Electronic systems are generally associated with an improved response rate and enhanced timeliness of responses in comparison with paper-based systems, without adversely affecting the quality of responses.