Clinical problem solving and social determinants of health: a descriptive study using unannounced standardized patients to directly observe how resident physicians respond to social determinants of health

Validity evidence for the clinical communication skills assessment tool (CCSAT) from 9 years of implementation in a high stakes medical student OSCE

OBJECTIVES

Communication and other clinical skills are routinely assessed in medical schools using Objective Structured Clinical Examinations (OSCEs) so routinely that it can be difficult to monitor and maintain validity. We report on the accumulation of validity evidence for the Clinical Communication Skills Assessment Tool (CCSAT) based on its use with 9 cohorts of medical students in a high stakes OSCE.

METHODS

We describe the implementation of the CCSAT including information on the underlying model, the tool's items, domains, scales and scoring, and its role in curriculum. Internal structure is explored through item, internal consistency, and confirmatory factor analyses. Evidence for CCSAT validity is synthesized within prevailing frameworks (Messick12 and Kane13) based on continuous quality improvement and use of the CCSAT for feedback, remediation, curricular design, and research.

RESULTS

Implementation of the CCSAT over time has facilitated our communication skills curriculum and training. Thoughtful case development and investment in standardized patient training has contributed to data quality. Item analysis supports our behaviorally anchored scale (not done, partly and well done) and the skills domains suggested by an a priori evidence-based clinical communication model were confirmed via analysis of actual student data. Evidence synthesized across the frameworks suggests consistent validity of the CCSAT for generalization inferences (that it captures the construct), responsiveness (sensitivity to change/difference), content validity/internal structure, relationships to other variables, and consequences/implications. More evidence is needed to strengthen validity of CCSAT scores for understanding extrapolation inferences and real-world implications.

CONCLUSIONS AND PRACTICE IMPLICATIONS

This pragmatic approach to evaluating validity within a program of assessment serves as a model for medical schools seeking to continuously monitor the quality of clinical skill assessments, a need made particularly relevant since the US NBME no longer requires the Step 2 Clinical Skills exam, leaving individual schools with the responsibility for ensuring graduates have acquired the requisite core clinical skills. We document strong evidence for CCSAT validity over time and across cohorts as well as areas for improvement and further examination.

Comparison of Primary Care Patients' and Unannounced Standardized Patients' Perceptions of Care

The objective of this study was to compare unannounced standardized patient (USP) and patient reports of care. Patient satisfaction surveys and USP checklist results collected at an urban, public hospital were compared to identify items included in both surveys. Qualitative commentary was reviewed to better understand USP and patient satisfaction survey data. Analyses included χ² and Mann-Whitney U test. Patients provided significantly higher ratings on 10 of the 11 items when compared to USPs. USPs may provide a more objective perspective on a clinical encounter than a real patient, reinforcing the notion that real patients skew overly positive or negative.

Experience of residents learning about social determinants of health and an assessment tool: Mixed-methods research

Background

Educating healthcare professionals about the social determinants of health is important in improving health outcomes of marginalized patients. Residents' experience of learning about the social determinants of health and a clinical assessment tool remains unclear.

Methods

Residents participated in an online session about the social determinants of health and the assessment tool. Using the New World Kirkpatrick Model, we obtained data about participants' experience from various perspectives. The data were analyzed using a concurrent triangulation mixed-methods design.

Results

The study included 20 out of 30 residents. Their response was good, and self-reported learning scores were increased by the session. They learned when to ask about patients' social conditions, what to ask, and how to coordinate medical care appropriately. Participants reported reflecting on their role as medical professionals and implementing new practices based on their learning, as well as concerns about addressing patients' social conditions.

Conclusion

Through learning about the social determinants of health, and assessment tools, residents both acquired knowledge and skills, and reflected on their previous practice, accepted patients as they are, understood difficult patients better, and developed interprofessional collaboration. Medical education about the social determinants of health can trigger changes in residents' views.