Monitoring communication skills progress of medical students: Establishing a baseline has value, predicting the future is difficult

Validity evidence for the clinical communication skills assessment tool (CCSAT) from 9 years of implementation in a high stakes medical student OSCE

OBJECTIVES

Communication and other clinical skills are routinely assessed in medical schools using Objective Structured Clinical Examinations (OSCEs) so routinely that it can be difficult to monitor and maintain validity. We report on the accumulation of validity evidence for the Clinical Communication Skills Assessment Tool (CCSAT) based on its use with 9 cohorts of medical students in a high stakes OSCE.

METHODS

We describe the implementation of the CCSAT including information on the underlying model, the tool's items, domains, scales and scoring, and its role in curriculum. Internal structure is explored through item, internal consistency, and confirmatory factor analyses. Evidence for CCSAT validity is synthesized within prevailing frameworks (Messick12 and Kane13) based on continuous quality improvement and use of the CCSAT for feedback, remediation, curricular design, and research.

RESULTS

Implementation of the CCSAT over time has facilitated our communication skills curriculum and training. Thoughtful case development and investment in standardized patient training has contributed to data quality. Item analysis supports our behaviorally anchored scale (not done, partly and well done) and the skills domains suggested by an a priori evidence-based clinical communication model were confirmed via analysis of actual student data. Evidence synthesized across the frameworks suggests consistent validity of the CCSAT for generalization inferences (that it captures the construct), responsiveness (sensitivity to change/difference), content validity/internal structure, relationships to other variables, and consequences/implications. More evidence is needed to strengthen validity of CCSAT scores for understanding extrapolation inferences and real-world implications.

CONCLUSIONS AND PRACTICE IMPLICATIONS

This pragmatic approach to evaluating validity within a program of assessment serves as a model for medical schools seeking to continuously monitor the quality of clinical skill assessments, a need made particularly relevant since the US NBME no longer requires the Step 2 Clinical Skills exam, leaving individual schools with the responsibility for ensuring graduates have acquired the requisite core clinical skills. We document strong evidence for CCSAT validity over time and across cohorts as well as areas for improvement and further examination.

Understanding medical student paths to communication skills expertise using latent profile analysis

PURPOSE

To describe patterns of clinical communication skills that inform curriculum enhancement and guide coaching of medical students.

MATERIALS AND METHODS

Performance data from 1182 consenting third year medical students in 9 cohorts (2011-2019), on a 17-item Clinical Communication Skills Assessment Tool (CCSAT) completed by trained Standardized Patients as part of an eight case high stakes Comprehensive Clinical Skills Exam (CCSE) were analyzed using latent profile analysis (LPA). Assessment domains included: information gathering (6 items), relationship development (5 items), patient education (3 items), and organization/time management (3 items). LPA clustered learners with similar strength/weakness into profiles based on item response patterns across cases. One-way analysis of variance (ANOVA) assessed for significant differences by profile for CCSAT items.

RESULTS

Student performance clustered into six profiles in three groups, high performing (HP1 and HP2-Low Patient Education, 15.7%), average performing (AP1 and AP2-Interrupters, 40.9%), and lower performing profiles (LP1-Non-interrupters and LP2, 43.4%) with adequate model fit estimations and similar distribution in each cohort. We identified 3 CCSAT items that discriminated among learner's skill profiles.

CONCLUSION

Clinical communication skill performance profiles provide nuanced, benchmarked guidance for curriculum improvement and tailoring of communication skills coaching.

Teaching SSHADESS Versus HEADSS to Medical Students: An Association With Improved Communication Skills and Increased Psychosocial Factor Assessments

OBJECTIVE

The most recent Bright Futures edition describes both the HEADSS (Home, Education, Drugs, Sexuality, Safety) and the strength-based SSHADESS (Strengths, School, Home, Activities, Drugs, Emotions, Sexuality, Safety) frameworks for conducting an adolescent psychosocial history. We found limited research comparing the effectiveness of these 2 frameworks. Our study objective was to examine whether teaching medical students SSHADESS versus HEADSS is associated with increased communication skills and/or completion of the psychosocial assessment.

METHODS

In this retrospective cohort study of pediatric clerkship students at New York University Grossman School of Medicine, we collected data from observer (faculty, fellow, and resident) and standardized patient (SP) assessments during Objective Structured Clinical Examinations. Primary outcomes were observer and SP-rated usage of communication skills. Secondary outcomes were observer-rated assessment of 6 psychosocial factors. Our predictor variable was whether students were taught HEADSS (11/2015-10/2016) or SSHADESS (11/2016-10/2017). We used Fisher's exact tests and then logistic regressions to adjust for pediatrics clerkship timing and baseline communication skills.

RESULTS

About 200 students were assessed (n = 97 HEADSS cohort, n = 103 SSHADESS cohort). In adjusted analyses of observer scores, the SSHADESS cohort was more likely to use all communication skills (adjusted odds ratio [aOR] 3.2, 95% confidence interval [CI] 1.7-6.3]) and assess all psychosocial factors (aOR 1.9, 95% CI 1.01-3.4). There was no significant difference in SP communication scores.

DISCUSSION

Teaching SSHADESS was associated with higher observer-rated communication skills scores and improved completeness in assessment of psychosocial factors. Future work should examine the efficacy of SSHADESS through workplace-based assessments and 360 degree assessments from adolescent patients.

Consensus on communication curriculum content in Danish undergraduate medical education: A Delphi study

BACKGROUND

The acquisition of skills in patient-centered communication is a critical aspect of medical education which demands both resource-intensive instruction and longitudinal opportunities for learning. Significant variation currently exists in the content and timing of communication education. The aim of this study was to establish consensus regarding communication curriculum content for undergraduate medical education (UME) within the country of Denmark.

METHODS

This study employed a Delphi process which is a widely accepted method for establishing consensus among experts and can be utilized to guide planning and decision-making in education. For this study, consensus was based on greater than 60% agreement between participants. Diverse stakeholders, representing all four universities with medical schools in Denmark, participated in an iterative three-round Delphi process which involved: (1) identifying key curricular elements for medical student education, (2) rating the importance of each item, and (3) prioritizing items relative to one another and rating each item based on the level of mastery that was expected for each skill (i.e. knowledge, performance with supervision, or performance independently).

RESULTS

A national sample of 149 stakeholders participated with a 70% response rate for round 1, 81% for round 2, and 86% for round 3. The completed Delphi process yielded 56 content items which were prioritized in rank order lists within five categories: (1) establishing rapport, engaging patient perspectives and responding to needs; (2) basic communication skills and techniques; (3) phases and structure of the encounter; (4) personal characteristics and skills of the student; (5) specific challenging patient groups and context-dependent situations.

DISCUSSION

Using a Delphi process, it was possible to achieve consensus regarding communication curriculum content for UME. These findings provide an important foundation for ensuring greater uniformity in UME, as well as supporting the important longitudinal goals of communication skill development across medical training.

Assessing how students value learning communication skills in an undergraduate anatomy and physiology course

Students, particularly those in science, technology, engineering and mathematics (STEM) and healthcare-related programs, should develop proficient interpersonal skills, including communication. To help students develop effective communication skills, instructors need to consider the value students give to learning these skills. The Student Attitudes Toward Communication Skills Survey (SATCSS) was developed to measure how undergraduate students value learning communication skills based on Expectancy-Value Theory across three modes of communication (verbal, written, non-verbal). The survey was given to students interested in healthcare professions and enrolled in an undergraduate anatomy and physiology (A&P) course (n = 233) at a Midwest research active university. The survey showed evidence of validity, measuring two components: (1) "Value to Profession" (attainment and utility value) and (2) "Value to Self" (intrinsic value and cost). There was a significant difference in sub-scores among the four task values such that students thought that learning communication skills was important and relevant (high attainment and utility value) but not interesting (low intrinsic value) and costly. Students with high total scores valued communication skills across all four task values. As total value scores decreased, it was first due to students finding learning communication skills to be time prohibitive and then a lack of interest in learning communication skills. Based on these results, it is recommended that instructors incorporate communication skills training into content that is already part of their A&P course to reduce time concerns. Additional recommendations include using reflective activities and humor to increase student interest.

A novel simulation-based approach to training for recruitment of older adults to clinical trials

BACKGROUND

The need to engage adults, age 65 and older, in clinical trials of conditions typical in older populations, (e.g. hypertension, diabetes mellitus, Alzheimer's disease and related dementia) is exponentially increasing. Older adults have been markedly underrepresented in clinical trials, often exacerbated by exclusionary study criteria as well as functional dependencies that preclude participation. Such dependencies may further exacerbate communication challenges. Consequently, the evidence of what works in subject recruitment is less generalizable to older populations, even more so for those from racial and ethnic minority and low-income communities.

METHODS

To support capacity of research staff, we developed a virtual, three station simulation (Group Objective Structured Clinical Experience-GOSCE) to teach research staff communication skills. This 2-h course included a discussion of challenges in recruiting older adults; skills practice with Standardized Participants (SPs) and faculty observer who provided immediate feedback; and debrief to highlight best practices. Each learner had opportunities for active learning and observational learning. Learners completed a retrospective pre-post survey about the experience. SP completed an 11-item communication checklist evaluating the learner on a series of established behaviorally anchored communication skills (29).

RESULTS

In the research staff survey, 92% reported the overall activity taught them something new; 98% reported it provided valuable feedback; 100% said they would like to participate again. In the SP evaluation there was significant variation: the percent well-done of items by case ranged from 25-85%.

CONCLUSIONS

Results from this pilot suggest that GOSCEs are a (1) acceptable; (2) low cost; and (3) differentiating mechanism for training and assessing research staff in communication skills and structural competency necessary for participant research recruitment.

Validation of the ComCare index for rater-based assessment of medical communication and interpersonal skills

OBJECTIVES

To develop and validate a short instrument to assess undergraduate medical students' communication and interpersonal skills in videographed history taking situations with simulated patients.

METHODS

Sixty-seven undergraduate medical students participating in an assessment including videographed physician-patient encounters for history taking with five simulated patients were included in this study. The last video of each participant's consultation hour was rated by two independent assessors with the eight-item ComCare index for assessment of communication and interpersonal skills newly designed for the external rater perspective (ComCareR). We compared the sum scores of the ComCareR with ratings of the same videos with the Kalamazoo Communication Skills Assessment Form from an observational perspective (KCSAFd-video) and the Global Rating scale (GR), which also measure communication and interpersonal skills.

RESULTS

The ComCareR showed an excellent interrater reliability (ICC = .85). We found a small but significant correlation with the KCSAFd-video Interpersonal Competence (ρ = .34, 95% CI [.10,.54]) and a high positive correlation with the GR (ρ = .59, 95% CI [.40,.73]).

CONCLUSIONS

The ComCareR is a valid and brief index for holistic assessment of communication and interpersonal skills in physician-patient encounters.

PRACTICE IMPLICATIONS

The ComCareR can be used for quick rater-based assessment of physicians' communication and interpersonal skills.

Teaching and Assessing Communication Skills in Pediatric Residents: How Do Parents Think We Are Doing?

OBJECTIVE

Curricula designed to teach and assess the communication skills of pediatric residents variably integrates the parent perspective. We compared pediatric residents' communication skills in an objective structured clinical exam (OSCE) case as assessed by Family Faculty (FF), parents of pediatric patients, versus standardized patients (SP).

METHODS

Residents participated in an OSCE case with a SP acting as a patient's parent. We compared resident performance as assessed by FF and SP with a behaviorally-anchored checklist. Items were rated as not done, partly done or well done, with well-done indicating mastery. The residents evaluated the experience.

RESULTS

42 residents consented to study participation. FF assessed a lower percentage of residents as demonstrating skill mastery as compared to SP in 19 of the 23 behaviors. There was a significant difference between FF and SP for Total Mastery Score and Mastery of the Competency Scores in three domains (Respect and Value, Information Sharing and Participation in Care and Decision Making). The majority of residents evaluated the experience favorably.

CONCLUSION

Involving parents of pediatric patients in the instructive and assessment components of a communication curriculum for pediatric residents adds a unique perspective and integrates the true stakeholders in parent-physician communication.

Does a measure of Medical Professional Identity Formation predict communication skills performance?

OBJECTIVE

To validate an approach to measuring professional identity formation (PIF), we explore if the Professional Identity Essay (PIE), a stage score measure of medical professional identity (PI), predicts clinical communication skills.

METHODS

Students completed the PIE during medical school orientation and a 3-case Objective Structured Clinical Exam (OSCE) where standardized patients reliably assessed communication skills in 5 domains. Using mediation analyses, relationships between PIE stage scores and communication skills were explored.

RESULTS

For the 351 (89%) consenting students, controlling for individual characteristics, there were increases in patient counseling (6.5%, p<0.01), information gathering (4.3%, p = 0.01), organization and management (4.1%, p = 0.02), patient assessment (3.6%, p = 0.04), and relationship development (3.5%, p = 0.03) skills for every half stage increase in PIE score. The communication skills of lower socio-economic status (SES) students are indirectly impacted by their slightly higher PIE stage scores.

CONCLUSION

Higher PIE stage scores are associated with higher communication skills and lower SES.

PRACTICE IMPLICATIONS

PIE predicts critical clinical skills and identifies how SES and other characteristics indirectly impact future clinical performance, providing validity evidence for using PIE as a tool in longitudinal formative academic coaching, program and curriculum evaluation, and research.

Effectiveness of patient education training on health professional student performance: A systematic review

BACKGROUND

Despite the fundamental role of patient education, concerns have been raised by health professionals, students, and professional bodies regarding student preparation for this area of practice.

OBJECTIVE

To evaluate the effectiveness of patient education training on health professional student performance.

METHOD

A systematic search was performed across PubMed, Cinahl, Embase, and Cochrane Library. Reference and forward citation searches of included articles were conducted. Inclusion criteria were published journal articles from 2010 to 2020 regarding patient education training for health professional students with measures relating to patient education performance. Quality appraisal of individual studies was conducted using Cochrane risk-of-bias v2 or ROBINS-I; overall appraisal was also determined. Sixteen articles met eligibility criteria. Independent data extraction was undertaken.

RESULTS

Fourteen studies reported improvements in student performance following patient education training; two studies reported no significant improvements. Of different modalities examined, simulation-based learning appeared to be most effective.

CONCLUSION

Patient education training enhances health professional students' performance of patient education. Future research is needed to evaluate effectiveness of pedagogically informed interventions across health professions using objective approaches to assess student performance in clinically relevant settings.

PRACTICAL VALUE

This review supports the use of specific training approaches to improve student performance of patient education.

Objective Structured Clinical Examinations as a Method of Competency Evaluation in a Primary Care Nurse Practitioner Program

BACKGROUND

Objective structured clinical examinations (OSCEs) are an effective method of instruction and evaluation. Their use in nurse practitioner (NP) programs as a means of evaluation warrants further exploration.

PURPOSE

The purpose of this project was to test the applicability and feasibility of implementation of OSCEs as a means of formative evaluation in a primary care NP program and to determine student feedback about the experience.

METHODS

This project evaluated the implementation of an educational intervention using 2 previously developed and tested OSCE scenarios with standardized patients.

RESULTS

Mean scores were computed for the subcategories, as well as for global rating scores on the OSCE. Strong correlations were found across categories within the scenario; no correlations were found between scenarios. Moderate intraclass correlation coefficients were achieved. Student feedback about the experience, including perceived competence and confidence, was overwhelmingly positive.

CONCLUSIONS

This project demonstrates success of a pilot OSCE program.

The TeleHealth OSCE: Preparing Trainees to Use Telemedicine as a Tool for Transitions of Care

BACKGROUND

Telemedicine holds promise to bridge the transition of care between inpatient and outpatient settings. Despite this, the unique communication and technical skills required for virtual encounters are not routinely taught or practiced in graduate medical education (GME) programs.

OBJECTIVE

To develop an objective structured clinical examination (OSCE) case to assess residents' telemedicine-specific skills and identify potential gaps in our residency program's curriculum.

METHODS

As part of a multi-station OSCE in 2019, we developed a case simulating a remote encounter between a resident and a recently discharged standardized patient. We developed an assessment tool comprising specific behaviors anchored to "not done," "partly done," and "well done" descriptors to evaluate core communication and telemedicine-specific skills.

RESULTS

Seventy-eight NYU internal medicine residents participated in the case. Evaluations from 100% of participants were obtained. Residents performed well in Information Gathering and Relationship Development domains. A mean 95% (SD 3.3%) and 91% (SD 4.9%) of residents received "well done" evaluations across these domains. A mean 78% (SD 14%) received "well done" within Education/Counseling domain. However, only 46% (SD 45%) received "well done" evaluations within the Telemedicine domain; specific weak areas included performing a virtual physical examination (18% well done) and leveraging video to augment history gathering (17% well done). There were no differences in telemedicine-specific skill evaluations when stratified by training track or postgraduate year.

CONCLUSIONS

We simulate a post-discharge virtual encounter and present a novel assessment tool that uncovers telemedicine-specific knowledge gaps in GME trainees.

Gasping for air: measuring patient education and activation skillsets in two clinical assessment contexts

Objective structured clinical examinations (OSCEs) provide a controlled, simulated setting for competency assessments, while unannounced simulated patients (USPs) measure competency in situ or real-world settings. This exploratory study describes differences in primary care residents’ skills when caring for the same simulated patient case in OSCEs versus in a USP encounter. Data reported describe a group of residents (n=20) who were assessed following interaction with the same simulated patient case in two distinct settings: an OSCE and a USP visit at our safety-net clinic from 2009 to 2010. In both scenarios, the simulated patient presented as an asthmatic woman with limited understanding of illness management. Residents were rated through a behaviourally anchored checklist on visit completion. Summary scores (mean % well done) were calculated by domain and compared using paired sample t-tests. Residents performed significantly better with USPs on 7 of 10 items and in two of three aggregate assessment domains (p<0.05). OSCE structure may impede assessment of activation and treatment planning skills, which are better assessed in real-world settings. This exploration of outcomes from our two assessments using the same clinical case lays a foundation for future research on variation in situated performance. Using both assessments during residency will provide a more thorough understanding of learner competency.

Igniting activation: Using unannounced standardized patients to measure patient activation in smoking cessation

Introduction

Despite a decline, smoking rates have remained high, especially in communities with lower income, education, and limited insurance options. Evidence shows that physician-initiated counseling on smoking cessation is effective and saves lives, and that specific skills are needed to appropriately lead this type of patient-physician communication. Residency is a critical moment for future physicians and may be the optimal time to learn, practice, and refine this skillset. Unannounced Standardized Patients (USPs) have been found to be effective, incognito evaluators of resident practices.

Methods

This study introduced rigorously trained actors (USPs) into two urban, safety-net clinics to assess resident ability to engage, activate, and counsel a pre-contemplative smoker. A complementary chart review assessed appropriate documentation in the patient's electronic health record (EHR) and its relationship to counseling style and prescribing practices.

Results

Resident scores (% well done) on patient education and engagement were low (33% and 23%, respectively). Residents who coupled cessation advice with an open discussion style activated their patients more than those who solely advised cessation across all comparable measures. On EHR documentation, residents who accurately documented smoking history were more likely to directly advise their patient to quit smoking when compared to residents who did not document (t(97) = 2.828, p = .006, Cohen's D = 0.56).

Conclusions

Results highlight the need to reinforce training in patient-centered approaches including motivational interviewing, counseling, and shared decision-making. Future research should focus on the effects of smokers in pre-contemplation on physician counseling style and examine the relationship between medical training and provider communication to guide interventions.

•

Counseling, documentation, and prescribing all vary when residents meet an Unannounced Standardized Patient (USP) smoker.

•

Patients are activated to quit smoking when a provider couples cessation advice with an open discussion of pros and cons.

•

Training in motivational interviewing and shared decision-making can enhance patient-provider cessation communication.