A Comparison of Assessment Tools: Is Direct Observation an Improvement Over Objective Structured Clinical Examinations for Communications Skills Evaluation?

Are evaluations in simulated medical encounters reliable among rater types? A comparison between standardized patient and outside observer ratings of OSCEs

Objective

By analyzing Objective Structured Clinical Examination (OSCE) evaluations of first-year interns' communication with standardized patients (SP), our study aimed to examine the differences between ratings of SPs and a set of outside observers with training in healthcare communication.

Methods

Immediately following completion of OSCEs, SPs evaluated interns' communication skills using 30 items. Later, two observers independently coded video recordings using the same items. We conducted two-tailed t-tests to examine differences between SP and observers' ratings.

Results

Rater scores differed significantly on 21 items (p < .05), with 20 of the 21 differences due to higher SP in-person evaluation scores. Items most divergent between SPs and observers included items related to empathic communication and nonverbal communication.

Conclusion

Differences between SP and observer ratings should be further investigated to determine if additional rater training is needed or if a revised evaluation measure is needed. Educators may benefit from adjusting evaluation criteria to decrease the number of items raters must complete and may do so by encompassing more global questions regarding various criteria. Furthermore, evaluation measures may be strengthened by undergoing reliability and validity testing.

Innovation

This study highlights the strengths and limitations to rater types (observers or SPs), as well as evaluation methods (recorded or in-person).

Effect of Experiential Communication Skills Education on Graduate Medical Education Trainees' Communication Behaviors: A Systematic Review

PURPOSE

A better understanding of how communication skills education impacts trainees' communication skills is important for continual improvement in graduate medical education (GME). Guided by the Kirkpatrick Model, this review focused on studies that measured communication skills in either simulated or clinical settings. The aim of this systematic review was to examine the effect of experiential communication skills education on GME trainees' communication behaviors.

METHOD

Five databases were searched for studies published between 2001 and 2021 using terms representing the concepts of medical trainees, communication, training, and skills and/or behaviors. Included studies had an intervention design, focused only on GME trainees as learners, used experiential methods, and had an outcome measure of communication skills behavior that was assessed by a simulated or standardized patient (SP), patient, family member, or outside observer. Studies were examined for differences in outcomes based on study design; simulated versus clinical evaluation setting; outside observer versus SP, patient, or family member evaluator; and length of training.

RESULTS

Seventy-seven studies were ultimately included. Overall, 54 (70%) studies reported some positive findings (i.e., change in behavior). There were 44 (57%) single-group pre-post studies, 13 (17%) nonrandomized control studies, and 20 (26%) randomized control studies. Positive findings were frequent in single-group designs (80%) and were likely in nonrandomized (62%) and randomized (55%) control trials. Positive findings were likely in studies evaluating communication behavior in simulated (67%) and clinical (78%) settings as well as in studies with outside observer (63%) and SP, patient, and family member (64%) evaluators.

CONCLUSIONS

This review demonstrates strong support that experiential communication skills education can impact GME trainees' communication behaviors. Marked heterogeneity in communication trainings and evaluation measures, even among subgroups, did not allow for meta-analysis or comparative efficacy evaluation of different studies. Future studies would benefit from homogeneity in curricular and evaluation measures.

Validation of the ComCare index for rater-based assessment of medical communication and interpersonal skills

OBJECTIVES

To develop and validate a short instrument to assess undergraduate medical students' communication and interpersonal skills in videographed history taking situations with simulated patients.

METHODS

Sixty-seven undergraduate medical students participating in an assessment including videographed physician-patient encounters for history taking with five simulated patients were included in this study. The last video of each participant's consultation hour was rated by two independent assessors with the eight-item ComCare index for assessment of communication and interpersonal skills newly designed for the external rater perspective (ComCareR). We compared the sum scores of the ComCareR with ratings of the same videos with the Kalamazoo Communication Skills Assessment Form from an observational perspective (KCSAFd-video) and the Global Rating scale (GR), which also measure communication and interpersonal skills.

RESULTS

The ComCareR showed an excellent interrater reliability (ICC = .85). We found a small but significant correlation with the KCSAFd-video Interpersonal Competence (ρ = .34, 95% CI [.10,.54]) and a high positive correlation with the GR (ρ = .59, 95% CI [.40,.73]).

CONCLUSIONS

The ComCareR is a valid and brief index for holistic assessment of communication and interpersonal skills in physician-patient encounters.

PRACTICE IMPLICATIONS

The ComCareR can be used for quick rater-based assessment of physicians' communication and interpersonal skills.

It does not have to be either or! Assessing competence in medicine should be a continuum between an analytic and a holistic approach

Assessing competence is a tremendous challenge in medical education. There are two contrasting approaches in competence assessment: an analytic approach that aims to precisely measure observable constituents and facets of competence and a holistic approach that focuses on a comprehensive assessment of competences in complex real situations reflecting actual performance. We would like to contribute to the existing discourse about medical competence and its assessment by proposing an approach that can provide orientation for the development of competence-based assessment concepts in undergraduate and postgraduate medical education. The approach follows Kane's framework of an "argument-based approach" to validity and is based on insights into task complexity, testing and learning theories as well as the importance of the learning environment. It describes a continuum from analytic to holistic approaches to assess the constituents and facets of competence to performance. We conclude that the complexity of a task should determine the selection of the assessment and suggest to use this approach to reorganize and adapt competence assessment.

The Enhanced Brief Structured Observation Model: Efficiently Assess Trainee Competence and Provide Feedback

Introduction

The regular observation of trainees is essential to ascertain trainee proficiency in competency-based assessments. Unfortunately, observation of residents is not frequent enough to facilitate entrustment decisions, and the busy clinician-educator may not have the tools or time to conduct effective and efficient observations.

Methods

We created a hands-on faculty development workshop utilizing an enhanced variation of the brief structured observation (BSO) technique to train both primary care and subspecialty pediatric faculty on how to effectively and efficiently observe trainees. The workshop has provided faculty a practical approach to observing trainees in a focused fashion and providing effective feedback on clinical skills based on their observation. In the workshop, faculty had an opportunity to observe residents taking an unrehearsed history from a medical student simulating an acutely ill patient, culminating in feedback on the residents' performance using the subjective, objective, assessment, and plan (SOAP) format.

Results

This faculty development workshop has been presented to more than 100 faculty both locally and nationally, and feedback has been uniformly positive, with three institutions incorporating this model into their programs to date.

Discussion

This enhanced BSO workshop promotes a model that streamlines the observations of trainees and provides faculty with the tools to encourage more observations.