Reliability of scores obtained from standardized patient and instructor assessments

Effects of virtual simulation on student pharmacists' ability to assess self-care patient cases

INTRODUCTION

The use of simulation is recognized as an effective means to teach skills necessary for pharmacy practice. It is essential these educational activities are evaluated. The purpose of this study is to evaluate the effectiveness of virtual simulation on student pharmacists' ability to assess and formulate a plan for patients seeking self-care.

METHODS

In a third-year course, student pharmacists are assessed on their abilities to assist a patient seeking self-care. This assessment is a standardized patient (SP) case graded using a checklist. Previously, students prepared by participating in workshops and lectures only. In 2017, students continued workshops and lecture, but MyDispense virtual exercises were also assigned. Student performance after using MyDispense was compared to the previous year's students. Each student was assigned one of six different cases for the SP assessment. The maximum possible score was 20. Statistics used were chi-square, Fischer's exact, and t-test.

RESULTS

Data from 135 students in the intervention group were compared to 178 students in the control group. No differences were detected between the mean total scores of all cases when the intervention group was compared to the control group (16.89 vs. 17.22, respectively). A significant difference was only detected for Case 4 (15.91 vs. 18.02, intervention vs. control respectively, P < .001). Significant differences on the grading checklist were identified for six items.

CONCLUSIONS

Virtual simulation may impact student pharmacists' ability to assist patients seeking self-care recommendations. However, not all changes identified were positive.

Implementing Standardized Patient Caregivers to Practice Difficult Conversations in a Pediatric Dentistry Course

INTRODUCTION

Standardized patient (SP) methodology has been used in health professional education to help students develop communication, deeper diagnostic reasoning, and critical thinking skills. Few examples demonstrate the use of SPs to practice difficult conversations with pediatric caregivers in the pediatric dentistry literature. The objective of this educational activity was to describe the implementation of three SPs in a pediatric dentistry course for second-year dental students.

METHODS

We developed three SP encounters covering interactions with caregivers of an infant with severe early childhood caries, an adolescent on the path to gender affirmation, and a child with autism and dental caries whose caregiver was resistant to fluoride- and silver-containing dental materials. We describe the case design process, rubric construction and calibration, student debriefing, and pandemic modifications. We evaluated the effectiveness of the implementation by thematic analysis of student reflections following each encounter using a qualitative descriptive framework.

RESULTS

Eighty-three students completed each encounter. Qualitative analysis showed that students preferred a more realistic encounter by having a child or other distraction present. Students relied on different elements of motivational interviewing depending on the objective of each encounter and the age of the patient. Overall, the SP encounters were well received by students and faculty as an alternative or supplement to traditional student evaluation methods.

DISCUSSION

We noted a number of lessons learned about implementing SP methodology in pediatric dental education. With these experiences now in place, future evaluations should measure student performance in the SP encounters against student performance during clinical care.

Training and Validation of Incognito Standardized Patients for Assessing Oncology Fellows’ Performance Regarding Breaking Bad News

Background: Standardized patient (SP) has been applied to measure learner’s communication challenges such as breaking bad news (BBN). When utilizing SP-based assessment, 2 steps should be considered in SP training; assessing SPs portrayal as the real patient (authenticity) and how SPs checklist fill out reproducibility. Objectives: In this study, we described the process of training authentic and consistent SPs for evaluating oncology fellows’ performance regarding BBN in Iran. Methods: In this cross-sectional study, 8 eligible SPs took part in a 3-day educational meeting. Four different scenarios were developed regarding cancer patients along with corresponding checklists representing common presentations of illness. The accuracy of SPs portrayal was evaluated by experts, using a previously validated rating scale during observation of their role-playing. The reproducibility of SPs’ portraits was measured, using a test-retest approach. The inter-rater agreement of the SPs’ ability to fill out the BBN scale was measured by comparing the correlation between the SPs, who completed the scale, and oncologist faculty members’ judgments, which is considered a gold standard. Results: The findings of this study indicated that the cut-off score for the SPs’ portrayal validity was 95%. The reliability of SPs portrayal was acceptable (r = 0.89). The inter-rater agreement between SPs and experts in filling the BBN scale (k = 0.82), as well as, the consistency of filling the BBN scale between SP groups were highly acceptable (k = 0.86). Conclusions: The present study has demonstrated that if SP is trained appropriately, they shave a high degree of reliability and validity to assess oncology fellows’ performance regarding BBN skills.