Differences in expectations of passing standards in communication skills for pre-clinical and clinical medical students

Validity Evidence Supporting the Comunicación y Habilidades Interpersonales (CAI) Scale for Medical Spanish Communication and Interpersonal Skills Assessment

PURPOSE

While many schools teach medical Spanish to improve communication with the growing Spanish-speaking population, a standardized assessment methodology is lacking. This study evaluates validity evidence for a newly developed Spanish Communication and Interpersonal Skills (CIS) scale.

METHOD

The authors developed the Comunicación y Habilidades Interpersonales (CAI, pronounced /kī/) scale by culturally and linguistically adapting a 14-item validated English CIS scale. Sources of validity evidence were gathered targeting content, response process, internal structure, relations to other variables, and consequences. The CAI was administered as part of an objective structured clinical examination at 2 medical schools from April to June 2020.

RESULTS

The authors mapped CAI items onto USMLE Step 2 CIS behavioral functions and medical Spanish learner competencies to examine content validity. The mean item difficulty and item discrimination estimates are 2.86 (SD = 0.34) and 0.81 (SD = 0.08), respectively, demonstrating good psychometric properties at the item level. Internal-consistency reliability for a single case was 0.96. Learner variance accounted for 12% of total variance, demonstrating ability to differentiate higher and lower performing learners; the person-case interaction accounted for 44% of total variance, indicating strong case specificity.

CONCLUSIONS

The CAI has higher case specificity than previously reported in English, suggesting that the nuanced contextual elements of individual cases may matter even more when learners are using limited language skills. Implementing the CAI more widely may promote inclusion of patient-centered communication skills as part of curricular content and activities. This study's validity evidence may inform the development of a future medical Spanish certification examination.

Evaluation of Communication Skills Among Physicians: A Systematic Review of Existing Assessment Tools

OBJECTIVE

The importance of physician training in communication skills for motivating patients to adopt a healthy life-style and optimize clinical outcomes is increasingly recognized. This study inventoried and systematically reviewed the psychometric properties of, and the skills assessed by, existing assessment tools used to evaluate communication skills among physicians.

METHODS

This review was conducted in accordance with the PRISMA guidelines (PROSPERO: CRD42018091932). Four databases (PUBMED, EMBASE, PsychINFO, and SCOPUS) were searched up to December 2018, generating 3902 unique articles, which were screened by two authors. A total of 57 articles met the inclusion criteria and underwent full data extraction.

RESULTS

Forty-five different assessment tools were identified. Only 47% of the studies mentioned underlying theories or models for designing the tool. Fifteen communication skills were assessed across the tools, the five most prevalent were information giving (46%) or gathering (40%), eliciting patients' perspectives (44%), planning/goal setting (37%), and closing the session (32%). Most tools (93%) assessed communication skills using in-person role play exercises with standardized (61%) or real (32%) patients, but only 54% described the expertise of the raters who performed the evaluations. Overall, reporting of the psychometric properties of the assessment tools was poor-moderate (4.5 ± 1.3 out of 9).

CONCLUSIONS

Despite identifying several existing physician communication assessment tools, a high degree of heterogeneity between these tools, in terms of skills assessed and study quality, was observed, and most have been poorly validated. Research is needed to rigorously develop and validate accessible, convenient, "user-friendly," and easy to administer and score communication assessment tools.

Validity evidence for Quality Improvement Knowledge Application Tool Revised (QIKAT-R) scores: consequences of rater number and type using neurology cases

OBJECTIVES

To develop neurology scenarios for use with the Quality Improvement Knowledge Application Tool Revised (QIKAT-R), gather and evaluate validity evidence, and project the impact of scenario number, rater number and rater type on score reliability.

METHODS

Six neurological case scenarios were developed. Residents were randomly assigned three scenarios before and after a quality improvement (QI) course in 2015 and 2016. For each scenario, residents crafted an aim statement, selected a measure and proposed a change to address a quality gap. Responses were scored by six faculty raters (two with and four without QI expertise) using the QIKAT-R. Validity evidence from content, response process, internal structure, relations to other variables and consequences was collected. A generalisability (G) study examined sources of score variability, and decision analyses estimated projected reliability for different numbers of raters and scenarios and raters with and without QI expertise.

RESULTS

Raters scored 163 responses from 28 residents. The mean QIKAT-R score was 5.69 (SD 1.06). G-coefficient and Phi-coefficient were 0.65 and 0.60, respectively. Interrater reliability was fair for raters without QI expertise (intraclass correlation = 0.53, 95% CI 0.30 to 0.72) and acceptable for raters with QI expertise (intraclass correlation = 0.66, 95% CI 0.02 to 0.88). Postcourse scores were significantly higher than precourse scores (6.05, SD 1.48 vs 5.22, SD 1.5; p < 0.001). Sufficient reliability for formative assessment (G-coefficient > 0.60) could be achieved by three raters scoring six scenarios or two raters scoring eight scenarios, regardless of rater QI expertise.

CONCLUSIONS

Validity evidence was sufficient to support the use of the QIKAT-R with multiple scenarios and raters to assess resident QI knowledge application for formative or low-stakes summative purposes. The results provide practical information for educators to guide implementation decisions.