Comparing narrative and multiple-choice formats in online communication skill assessment

What do we know about written assessment of health professionals' communication skills? A scoping review

OBJECTIVE

The aim of this scoping review was to investigate the published literature on written assessment of communication skills in health professionals' education.

METHODS

Pubmed, Embase, Cinahl and Psychnfo were screened for the period 1/1995-7/2020. Selection was conducted by four pairs of reviewers. Four reviewers extracted and analyzed the data regarding study, instrument, item, and psychometric characteristics.

RESULTS

From 20,456 assessed abstracts, 74 articles were included which described 70 different instruments. Two thirds of the studies used written assessment to measure training effects, the others focused on the development/validation of the instrument. Instruments were usually developed by the authors, often with little mention of the test development criteria. The type of knowledge assessed was rarely specified. Most instruments included clinical vignettes. Instrument properties and psychometric characteristics were seldom reported.

CONCLUSION

There are a number of written assessments available in the literature. However, the reporting of the development and psychometric properties of these instruments is often incomplete. Practice implications written assessment of communication skills is widely used in health professions education. Improvement in the reporting of instrument development, items and psychometrics may help communication skills teachers better identify when, how and for whom written assessment of communication should be used.

Evaluation of Communication Skills Among Physicians: A Systematic Review of Existing Assessment Tools

OBJECTIVE

The importance of physician training in communication skills for motivating patients to adopt a healthy life-style and optimize clinical outcomes is increasingly recognized. This study inventoried and systematically reviewed the psychometric properties of, and the skills assessed by, existing assessment tools used to evaluate communication skills among physicians.

METHODS

This review was conducted in accordance with the PRISMA guidelines (PROSPERO: CRD42018091932). Four databases (PUBMED, EMBASE, PsychINFO, and SCOPUS) were searched up to December 2018, generating 3902 unique articles, which were screened by two authors. A total of 57 articles met the inclusion criteria and underwent full data extraction.

RESULTS

Forty-five different assessment tools were identified. Only 47% of the studies mentioned underlying theories or models for designing the tool. Fifteen communication skills were assessed across the tools, the five most prevalent were information giving (46%) or gathering (40%), eliciting patients' perspectives (44%), planning/goal setting (37%), and closing the session (32%). Most tools (93%) assessed communication skills using in-person role play exercises with standardized (61%) or real (32%) patients, but only 54% described the expertise of the raters who performed the evaluations. Overall, reporting of the psychometric properties of the assessment tools was poor-moderate (4.5 ± 1.3 out of 9).

CONCLUSIONS

Despite identifying several existing physician communication assessment tools, a high degree of heterogeneity between these tools, in terms of skills assessed and study quality, was observed, and most have been poorly validated. Research is needed to rigorously develop and validate accessible, convenient, "user-friendly," and easy to administer and score communication assessment tools.

Using an Audience Response System Smartphone App to Improve Resident Education in the Pediatric Intensive Care Unit

In the Pediatric Intensive Care Unit (PICU), most teaching occurs during bedside rounds, but technology now provides new opportunities to enhance education. Specifically, smartphone apps allow rapid communication between instructor and student. We hypothesized that using an audience response system (ARS) app can identify resident knowledge gaps, guide teaching, and enhance education in the PICU. Third-year pediatric residents rotating through the PICU participated in ARS-based education or received traditional teaching. Before rounds, experimental subjects completed an ARS quiz using the Socrative app. Concomitantly, the fellow leading rounds predicted quiz performance. Then, discussion points based on the incorrect answers were used to guide instruction. Scores on the pre-rotation test were similar between groups. On the post-rotation examination, ARS participants did not increase their scores more than controls. The fellow's prediction of performance was poor. Residents felt that the method enhanced their education whereas fellows reported that it improved their teaching efficiency. Although there was no measurable increase in knowledge using the ARS app, it may still be a useful tool to rapidly assess learners and help instructors provide learner-centered education.

Sexual Health in Undergraduate Medical Education: Existing and Future Needs and Platforms

INTRODUCTION

This article explores the evolution and current delivery of undergraduate medical education in human sexuality.

AIM

To make recommendations regarding future educational needs, principles of curricular development, and how the International Society for Sexual Medicine (ISSM) should address the need to enhance and promote human sexuality education around the world.

METHODS

The existing literature was reviewed for sexuality education, curriculum development, learning strategies, educational formats, evaluation of programs, evaluation of students, and faculty development.

MAIN OUTCOME MEASURES

The prevailing theme of most publications in this vein is that sexuality education in undergraduate medical education is currently not adequate to prepare students for future practice.

RESULTS

We identified components of the principles of attitudes, knowledge, and skills that should be contained in a comprehensive curriculum for undergraduate medical education in human sexuality. Management of sexual dysfunction; lesbian, gay, bisexual, and transgender health care; sexuality across genders and lifespan; understanding of non-normative sexual practices; sexually transmitted infections and HIV, contraception; abortion; sexual coercion and violence; and legal aspects were identified as topics meriting particular attention.

CONCLUSION

Curricula should be integrated throughout medical school and based on principles of adult learning. Methods of teaching should be multimodal and evaluations of student performance are critical. To realize much of what needs to be done, faculty development is critical. Thus, the ISSM can play a key role in the provision and dissemination of learning opportunities and materials, it can promote educational programs around the world, and it can articulate a universal curriculum with modules that can be adopted. The ISSM can create chapters, review documents, slide decks, small group and roleplay topics, and video-recorded materials and make all this material easily available. An expert consensus conference would be needed to realize these recommendations and fulfill them.

Development and validation of a computer-based situational judgement test to assess medical students' communication skills in the field of shared decision making

OBJECTIVE

To develop a computer-based test (CBT) measuring medical students' communication skills in the field of shared decision making (SDM) and to evaluate its construct validity.

METHODS

The CBT was developed in the context of an experimental study comparing three different trainings for SDM (including e-learning and/or role-play) and a control group. Assessment included a CBT (Part A: seven context-poor questions, Part B: 15 context-rich questions) and interviews with two simulated patients (SP-assessment). Cronbach's α was used to test the internal consistency. Correlations between CBT and SP-assessment were used to further evaluate construct validity of the CBT.

RESULTS

Seventy-two students took part in the study. Mean value for the CBT score was 72% of the total score. Cronbach's α was 0.582. After eliminating three items, Cronbach's α increased to 0.625. Correlations between the CBT and SP-assessment were low to moderate. The control group scored significantly lower than the training settings (p<0.001).

CONCLUSION

The CBT was reliable enough to test for group differences. For summative assessment purposes, considerably more questions would be needed.

PRACTICE IMPLICATIONS

We encourage teachers who particularly work with large student numbers to consider CBT as a feasible assessment method for cognitive aspects of communication skills.

Tools to Assess Behavioral and Social Science Competencies in Medical Education: A Systematic Review

PURPOSE

Behavioral and social science (BSS) competencies are needed to provide quality health care, but psychometrically validated measures to assess these competencies are difficult to find. Moreover, they have not been mapped to existing frameworks, like those from the Liaison Committee on Medical Education (LCME) and Accreditation Council for Graduate Medical Education (ACGME). This systematic review aimed to identify and evaluate the quality of assessment tools used to measure BSS competencies.

METHOD

The authors searched the literature published between January 2002 and March 2014 for articles reporting psychometric or other validity/reliability testing, using OVID, CINAHL, PubMed, ERIC, Research and Development Resource Base, SOCIOFILE, and PsycINFO. They reviewed 5,104 potentially relevant titles and abstracts. To guide their review, they mapped BSS competencies to existing LCME and ACGME frameworks. The final included articles fell into three categories: instrument development, which were of the highest quality; educational research, which were of the second highest quality; and curriculum evaluation, which were of lower quality.

RESULTS

Of the 114 included articles, 33 (29%) yielded strong evidence supporting tools to assess communication skills, cultural competence, empathy/compassion, behavioral health counseling, professionalism, and teamwork. Sixty-two (54%) articles yielded moderate evidence and 19 (17%) weak evidence. Articles mapped to all LCME standards and ACGME core competencies; the most common was communication skills.

CONCLUSIONS

These findings serve as a valuable resource for medical educators and researchers. More rigorous measurement validation and testing and more robust study designs are needed to understand how educational strategies contribute to BSS competency development.

A review of creative and expressive writing as a pedagogical tool in medical education

CONTEXT

The act of writing offers an opportunity to foster self-expression and organisational abilities, along with observation and descriptive skills. These soft skills are relevant to clinical thinking and medical practice. Medical school curricula employ pedagogical approaches suitable for assessing medical and clinical knowledge, but teaching methods for soft skills in critical thinking, listening and verbal expression, which are important in patient communication and engagement, may be less formal. Creative and expressive writing that is incorporated into medical school courses or clerkships offers a vehicle for medical students to develop soft skills. The aim of this review was to explore creative and expressive writing as a pedagogical tool in medical schools in relation to outcomes of medical education.

METHODS

This project employed a scoping review approach to gather, evaluate and synthesise reports on the use of creative and expressive writing in US medical education. Ten databases were searched for scholarly articles reporting on creative or expressive writing during medical school. Limitation of the results to activities associated with US medical schools, produced 91 articles. A thematic analysis of the articles was conducted to identify how writing was incorporated into the curriculum.

RESULTS

Enthusiasm for writing as a pedagogical tool was identified in 28 editorials and overviews. Quasi-experimental, mixed methods and qualitative studies, primarily writing activities, were aimed at helping students cognitively or emotionally process difficult challenges in medical education, develop a personal identity or reflect on interpersonal skills. The programmes and interventions using creative or expressive writing were largely associated with elective courses or clerkships, and not required courses.

CONCLUSIONS

Writing was identified as a potentially relevant pedagogical tool, but not included as an essential component of medical school curricula.

Implementation outcomes of a multiinstitutional web-based ethical, legal, and social implications genetics curriculum for primary care residents in three specialties

PURPOSE

Medical genetics lends itself to disseminated teaching methods because of mismatches between numbers of physicians having patients with genetic disorders and availability of genetic specialists.

METHOD

During 3 years, we implemented an interactive, web-based curriculum on ethical, legal, and social implications in medical genetics for primary care residents in three specialties at three institutions. Residents took five (of 10) cases and three (of five) tutorials that varied by specialty. We assessed changes in self-efficacy (primary outcome), knowledge, application, and viewpoints.

RESULTS

Overall enrollment was 69% (279/403). One institution did not complete implementation and was dropped from pre-post comparisons. We developed a six-factor ethical, legal, and social implications self-efficacy scale (Cronbach α = 0.95). Baseline self-efficacy was moderate (71/115; range: 23-115) and increased 15% after participation. Pre-post knowledge scores were high and unchanged. Residents reported that this curriculum covered ethical, legal, and social implications/genetics better than their usual curricula. Most (68-91%) identified advantages, especially in providing flexibility and stimulating self-directed learning. After participation, residents reported creating learning goals (66%) and acting on those goals (62%).

CONCLUSIONS

Ethical, legal, and social implications genetics curricular participation led to modest self-efficacy gains. Residents reported that the curriculum covered unique content areas, had advantages over traditional curriculum, and that they applied ethical, legal, and social implications content clinically. We share lessons from developing and implementing this complex web-based curriculum across multiple institutions.

The feasibility of a multi-format Web-based assessment of physicians' communication skills

OBJECTIVE

Little is known about the best approaches and format for measuring physicians' communication skills in an online environment. This study examines the reliability and validity of scores from two Web-based communication skill assessment formats.

METHODS

We created two online communication skill assessment formats: (a) MCQ (multiple-choice questions) consisting of video-based multiple-choice questions; (b) multi-format including video-based multiple-choice questions with rationales, Likert-type scales, and free text responses of what physicians would say to a patient. We randomized 100 general internists to each test format. Peer and patient ratings collected via the American Board of Internal Medicine (ABIM) served as validity sources.

RESULTS

Seventy-seven internists completed the tests (MCQ: 38; multi-format: 39). The adjusted reliability was 0.74 for both formats. Excellent communicators, as based on their peer and patient ratings, performed slightly better on both tests than adequate communicators, though this difference was not statistically significant. Physicians in both groups rated test format innovative (4.2 out of 5.0).

CONCLUSION

The acceptable reliability and participants' overall positive experiences point to the value of ongoing research into rigorous Web-based communication skills assessment.

PRACTICE IMPLICATIONS

With efficient and reliable scoring, the Web offers an important way to measure and potentially enhance physicians' communication skills.