1
|
Ark T, Kalet A, Tewksbury L, Altshuler L, Crowe R, Wilhite J, Hardowar K, Zabar S, Gillespie C. Validity evidence for the clinical communication skills assessment tool (CCSAT) from 9 years of implementation in a high stakes medical student OSCE. PATIENT EDUCATION AND COUNSELING 2024; 127:108323. [PMID: 38851013 DOI: 10.1016/j.pec.2024.108323] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/08/2023] [Revised: 03/22/2024] [Accepted: 05/13/2024] [Indexed: 06/10/2024]
Abstract
OBJECTIVES Communication and other clinical skills are routinely assessed in medical schools using Objective Structured Clinical Examinations (OSCEs) so routinely that it can be difficult to monitor and maintain validity. We report on the accumulation of validity evidence for the Clinical Communication Skills Assessment Tool (CCSAT) based on its use with 9 cohorts of medical students in a high stakes OSCE. METHODS We describe the implementation of the CCSAT including information on the underlying model, the tool's items, domains, scales and scoring, and its role in curriculum. Internal structure is explored through item, internal consistency, and confirmatory factor analyses. Evidence for CCSAT validity is synthesized within prevailing frameworks (Messick12 and Kane13) based on continuous quality improvement and use of the CCSAT for feedback, remediation, curricular design, and research. RESULTS Implementation of the CCSAT over time has facilitated our communication skills curriculum and training. Thoughtful case development and investment in standardized patient training has contributed to data quality. Item analysis supports our behaviorally anchored scale (not done, partly and well done) and the skills domains suggested by an a priori evidence-based clinical communication model were confirmed via analysis of actual student data. Evidence synthesized across the frameworks suggests consistent validity of the CCSAT for generalization inferences (that it captures the construct), responsiveness (sensitivity to change/difference), content validity/internal structure, relationships to other variables, and consequences/implications. More evidence is needed to strengthen validity of CCSAT scores for understanding extrapolation inferences and real-world implications. CONCLUSIONS AND PRACTICE IMPLICATIONS This pragmatic approach to evaluating validity within a program of assessment serves as a model for medical schools seeking to continuously monitor the quality of clinical skill assessments, a need made particularly relevant since the US NBME no longer requires the Step 2 Clinical Skills exam, leaving individual schools with the responsibility for ensuring graduates have acquired the requisite core clinical skills. We document strong evidence for CCSAT validity over time and across cohorts as well as areas for improvement and further examination.
Collapse
Affiliation(s)
- Tavinder Ark
- Medical College of Wisconsin, 8701 Watertown Plank Road, Milwaukee, WI 53226, USA
| | - Adina Kalet
- Medical College of Wisconsin, 8701 Watertown Plank Road, Milwaukee, WI 53226, USA
| | - Linda Tewksbury
- NYU Grossman School of Medicine, NYU Langone Health, 462 1st Avenue, New York, NY 10016, USA
| | - Lisa Altshuler
- NYU Grossman School of Medicine, NYU Langone Health, 462 1st Avenue, New York, NY 10016, USA
| | - Ruth Crowe
- NYU Long Island School of Medicine, NYU Langone Health, 101 Mineola Blvd, Mineola, NY 11501, USA
| | - Jeffrey Wilhite
- NYU Grossman School of Medicine, NYU Langone Health, 462 1st Avenue, New York, NY 10016, USA.
| | - Khemraj Hardowar
- NYU Grossman School of Medicine, NYU Langone Health, 462 1st Avenue, New York, NY 10016, USA
| | - Sondra Zabar
- NYU Grossman School of Medicine, NYU Langone Health, 462 1st Avenue, New York, NY 10016, USA
| | - Colleen Gillespie
- NYU Grossman School of Medicine, NYU Langone Health, 462 1st Avenue, New York, NY 10016, USA; Institute for Innovations in Medical Education, NYU Grossman School of Medicine, 462 1st Avenue, New York, NY 10016, USA
| |
Collapse
|
2
|
Altshuler L, Fisher H, Wilhite J, Phillips Z, Holmes I, Greene RE, Wallach AB, Smith R, Hanley K, Schwartz MD, Zabar S. Comparison of Primary Care Patients' and Unannounced Standardized Patients' Perceptions of Care. J Patient Exp 2023; 10:23743735231158940. [PMID: 36865378 PMCID: PMC9972044 DOI: 10.1177/23743735231158940] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/02/2023] Open
Abstract
The objective of this study was to compare unannounced standardized patient (USP) and patient reports of care. Patient satisfaction surveys and USP checklist results collected at an urban, public hospital were compared to identify items included in both surveys. Qualitative commentary was reviewed to better understand USP and patient satisfaction survey data. Analyses included χ2 and Mann-Whitney U test. Patients provided significantly higher ratings on 10 of the 11 items when compared to USPs. USPs may provide a more objective perspective on a clinical encounter than a real patient, reinforcing the notion that real patients skew overly positive or negative.
Collapse
Affiliation(s)
- Lisa Altshuler
- Division of General Internal Medicine and Clinical Innovation, NYU
Grossman School of Medicine, New York, NY, USA
| | - Harriet Fisher
- Division of General Internal Medicine and Clinical Innovation, NYU
Grossman School of Medicine, New York, NY, USA
| | - Jeffrey Wilhite
- Division of General Internal Medicine and Clinical Innovation, NYU
Grossman School of Medicine, New York, NY, USA
| | - Zoe Phillips
- Division of General Internal Medicine and Clinical Innovation, NYU
Grossman School of Medicine, New York, NY, USA,Lisa Altshuler, Division of General
Internal Medicine and Clinical Innovation, NYU Grossman School of Medicine, 462
First Avenue, OBV-CD401, New York, NY 10016, USA.
| | - Isaac Holmes
- Adult Primary Care Clinic, NYC Health +
Hospitals, New York, NY, USA
| | - Richard E Greene
- Division of General Internal Medicine and Clinical Innovation, NYU
Grossman School of Medicine, New York, NY, USA,Adult Primary Care Clinic, NYC Health +
Hospitals, New York, NY, USA
| | - Andrew B Wallach
- Division of General Internal Medicine and Clinical Innovation, NYU
Grossman School of Medicine, New York, NY, USA,Adult Primary Care Clinic, NYC Health +
Hospitals, New York, NY, USA
| | - Reina Smith
- Adult Primary Care Clinic, NYC Health +
Hospitals, New York, NY, USA
| | - Kathleen Hanley
- Division of General Internal Medicine and Clinical Innovation, NYU
Grossman School of Medicine, New York, NY, USA,Adult Primary Care Clinic, NYC Health +
Hospitals, New York, NY, USA
| | - Mark D Schwartz
- Division of General Internal Medicine and Clinical Innovation, NYU
Grossman School of Medicine, New York, NY, USA,Department of Population Health, NYU Grossman School of Medicine,
New York, NY, USA
| | - Sondra Zabar
- Division of General Internal Medicine and Clinical Innovation, NYU
Grossman School of Medicine, New York, NY, USA,Adult Primary Care Clinic, NYC Health +
Hospitals, New York, NY, USA
| |
Collapse
|
3
|
Mizumoto J, Son D, Izumiya M, Horita S, Eto M. Experience of residents learning about social determinants of health and an assessment tool: Mixed-methods research. J Gen Fam Med 2022; 23:319-326. [PMID: 36093216 PMCID: PMC9444009 DOI: 10.1002/jgf2.559] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/15/2022] [Revised: 05/02/2022] [Accepted: 05/06/2022] [Indexed: 12/04/2022] Open
Abstract
Background Educating healthcare professionals about the social determinants of health is important in improving health outcomes of marginalized patients. Residents' experience of learning about the social determinants of health and a clinical assessment tool remains unclear. Methods Residents participated in an online session about the social determinants of health and the assessment tool. Using the New World Kirkpatrick Model, we obtained data about participants' experience from various perspectives. The data were analyzed using a concurrent triangulation mixed-methods design. Results The study included 20 out of 30 residents. Their response was good, and self-reported learning scores were increased by the session. They learned when to ask about patients' social conditions, what to ask, and how to coordinate medical care appropriately. Participants reported reflecting on their role as medical professionals and implementing new practices based on their learning, as well as concerns about addressing patients' social conditions. Conclusion Through learning about the social determinants of health, and assessment tools, residents both acquired knowledge and skills, and reflected on their previous practice, accepted patients as they are, understood difficult patients better, and developed interprofessional collaboration. Medical education about the social determinants of health can trigger changes in residents' views.
Collapse
Affiliation(s)
- Junki Mizumoto
- Department of Medical Education Studies, International Research Center for Medical Education, Graduate School of MedicineThe University of TokyoTokyoJapan
| | - Daisuke Son
- Department of Community‐based Family Medicine, School of MedicineTottori University Faculty of MedicineYonagoJapan
| | - Masashi Izumiya
- Department of Medical Education Studies, International Research Center for Medical Education, Graduate School of MedicineThe University of TokyoTokyoJapan
| | - Shoko Horita
- Department of Medical Education Studies, International Research Center for Medical Education, Graduate School of MedicineThe University of TokyoTokyoJapan
| | - Masato Eto
- Department of Medical Education Studies, International Research Center for Medical Education, Graduate School of MedicineThe University of TokyoTokyoJapan
| |
Collapse
|
4
|
Graber ML. Progress understanding diagnosis and diagnostic errors: thoughts at year 10. Diagnosis (Berl) 2020; 7:151-159. [DOI: 10.1515/dx-2020-0055] [Citation(s) in RCA: 9] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/23/2020] [Accepted: 05/02/2020] [Indexed: 01/12/2023]
Affiliation(s)
- Mark L. Graber
- Society to Improve Diagnosis in Medicine , Evanston , IL , USA
| |
Collapse
|