Training Standardized Patients for a High-Stakes Clinical Performance Examination in the California Consortium for the Assessment of Clinical Competence

Development of a validated assessment tool for medical students using simulated patients: an 8-year panel survey

BACKGROUND

The use of simulated patients (SPs) to assess medical students' clinical performance is gaining prominence, underscored by patient safety perspective. However, few reports have investigated the validity of such assessment. Here, we examined the validity and reliability of an assessment tool that serves as a standardized tool for SPs to assess medical students' medical interview.

METHODS

This longitudinal survey was conducted at Keio University School of Medicine in Japan from 2014 to 2021. To establish content validity, the simulated patient assessment tool (SPAT) was developed by several medical education specialists from 2008 to 2013. A cohort of 36 SPs assessed the performance of 831 medical students in clinical practice medical interview sessions from April 2014 to December 2021. The assessment's internal structure was analyzed using descriptive statistics (maximum, minimum, median, mean, and standard deviation) for the SPAT's 13 item total scores. Structural validity was examined with exploratory factor analysis, and internal consistency with Cronbach's alpha coefficients. The mean SPAT total scores across different SPs and scenarios were compared using one way analysis of variance (ANOVA). Convergent validity was determined by correlating SPAT with the post-clinical clerkship obstructive structured clinical examination (post-CC OSCE) total scores using Pearson's correlation coefficient.

RESULTS

Of the 831 assessment sheets, 36 with missing values were excluded, leaving 795 for analysis. Thirty-five SPs, excluding one SP who quit in 2014, completed 795 assessments, for a response rate of 95.6%. Exploratory factor analysis revealed two factors, communication and physician performance. The overall Cronbach's alpha coefficient was 0.929. Significant differences in SPAT total scores were observed across SPs and scenarios via one-way ANOVA. A moderate correlation (r =.212, p <.05) was found between SPAT and post-CC OSCE total scores, indicating convergent validity.

CONCLUSIONS

Evidence for the validity of SPAT was examined. These findings may be useful in the standardization of SP assessment of the scenario-based clinical performance of medical students.

After the Discontinuation of Step 2 CS: A Collaborative Statement from the Directors of Clinical Skills Education (DOCS)

Issue: The United States Medical Licensing Exam (USMLE) Step 2 Clinical Skills Examination (Step 2 CS), the only clinical skills competency testing required for licensure in the United States, has been discontinued. Evidence: This exam, though controversial, propelled a movement emphasizing the value of clinical skills instruction and assessment in undergraduate medical education. While disappointed by the loss of this national driver that facilitated standardization of clinical skills education, the Directors of Clinical Skills Education (DOCS) see prospects for educational innovation and growth. DOCS is a national organization and inclusive community of clinical skills education leaders. This statement from DOCS regarding the discontinuation of USMLE Step 2 CS has been informed by DOCS meetings, listserv discussions, an internal survey, and a review of recent literature. Implications: Rigorous clinical skills assessment remains central to effective and patient-centered healthcare. DOCS shares specific concerns as well as potential solutions. Now free from the external pressure to prepare students for success on Step 2 CS, clinical skills educators can reprioritize content and restructure clinical skills programs to best meet the needs of learners and the ever-evolving healthcare landscape. DOCS, as an organization of clinical skills leaders, makes the following recommendations: 1) Collaboration amongst institutions must be prioritized; clinical skills assessment consortia should be expanded. 2) Governing, accrediting, and licensing organizations should leverage their influence to support and require high quality clinical skills assessments. 3) UME clinical skills leaders should develop ways to identify students who perform with exceptional, borderline, and poor clinical skills at their local institutions. 4) UME leadership should fully commit resources and curricular time to graduate students with excellent clinical skills.

A Call to Action From the California Consortium for the Assessment of Clinical Competence: Making the Case for Regional Collaboration

The discontinuation of the United States Medical Licensing Examination Step 2 Clinical Skills Examination emphasizes the need for other reliable standardized assessments of medical student clinical skills. For 30 years, the California Consortium for the Assessment of Clinical Competence (CCACC) has collaborated in the development of clinical skills assessments and has become a valuable resource for clinicians, standardized patient educators, psychometricians, and medical educators. There are many merits to strong multi-institutional partnerships, including the integration of data across multiple schools to provide feedback to both students and curricula, pooled test performance statistics for analysis and quality assurance, shared best practices and resources, individual professional development, and opportunities for research and scholarship. The infrastructure of the CCACC allows member schools to adapt to a changing medical landscape, from emerging trends in clinical medicine to the limitations imposed by a global pandemic. In the absence of a national examination, there is now a greater need for all medical schools to develop a comprehensive, dynamic, and psychometrically sound assessment that accurately evaluates clinical skills. Medical schools working together in regional consortia have the opportunity to create and implement innovative and robust assessments that evaluate a wide range of clinical skills, ensure that medical students have met an expected level of clinical competency before graduation, and provide a framework that contributes to ongoing efforts for the development of new national clinical skills standards.

Who Can Be the Standardized Patient in Disaster Medical Full-Scale Exercise, Freshman, or Professional?

BACKGROUND

Disasters cannot be predicted, so being well-prepared is important before a disaster strike. Good preparation needs constant practice and improvement, and full-scale exercise can provide both. The standardized patients (SPs) in a full-scale exercise to simulate patients are vital because they can provide realistic effects to exercise participants. However, there was no literature about who is capable of being an SP. We investigated the relationship between the SPs' current occupation and previous experience with their fidelity and participants' performance.

METHODS

Three identically designed full-scale exercises were conducted for three different emergency medical teams (EMTs) with the scenario of post-earthquake mass casualty incidents. Forty SPs were used in each exercise. Exercise objective and detail scenario were told to the SPs before exercise, and mock wound makeup was applied to the SPs. Each SP's occupation and previous experience were recorded before exercise. The SP's previous experience was defined as previous exercise experience and previous disaster medicine education. The SPs' fidelity (SPF) was rated using a 5-point Likert scale (1 = poor and 5 = excellent). The participants' performance (PP) was also rated using a 5-point Likert scale (1 = poor and 5 = excellent) according to the accuracy of each SP's triage result and management. The SPF and the PP were evaluated by clinicians with disaster medicine specialties using the same standard. The relationship between the SPs' occupation and the SPF along with occupation, and the PP were analyzed by analysis of variance (ANOVA). The data of the SPF and the PP in the SP groups divided by previous experience were analyzed by the Student's t -test.

RESULTS

The SPs' occupations were medical student, nurse, physician, and members of EMT. There was no significant difference in the SPF and the PP between occupations ( p = 0.382 and 0.416, respectively). Both the experienced and the inexperienced SP groups show no significant difference in the SPF ( p = 0.339). Significantly better PP was noted in the inexperienced SP group ( p < 0.001).

CONCLUSIONS

The SPs' background does not influence the performance of either SPs or exercise participants. We proposed that the success of using freshmen as SPs in full-scale exercise depends on the pre-exercise design and the SP instruction and training.

Training for Failure: A Simulation Program for Emergency Medicine Residents to Improve Communication Skills in Service Recovery

OBJECTIVES

Service failures such as long waits, testing delays, and medical errors are daily occurrences in every emergency department (ED). Service recovery refers to the immediate response of an organization or individual to resolve these failures. Effective service recovery can improve the experience of both the patient and the physician. This study investigated a simulation-based program to improve service recovery skills in postgraduate year 1 emergency medicine (PGY-1 EM) residents.

METHODS

Eighteen PGY-1 EM residents participated in six cases that simulated common ED service failures. The patient instructors (PIs) participating in each case and two independent emergency medicine (EM) faculty observers used the modified Master Interview Rating Scale to assess the communication skills of each resident in three simulation cases before and three simulation cases after a service recovery debriefing. For each resident, the mean scores of the first three cases and those of the last three cases were termed pre- and postintervention scores, respectively. The means and standard deviations of the pre- and postintervention scores were calculated by the type of rater and compared using paired t-tests. Additionally, the mean scores of each case were summarized. In the framework of the linear mixed-effects model, the variance in scores from the PIs and faculty observers was decomposed into variance contributed by PIs/cases, the program effect on individual residents, and the unexplained variance. In reliability analyses, the intraclass correlation coefficient between rater types and the 95% confidence interval were reported before and after the intervention.

RESULTS

When rated by the PIs, the pre- and postintervention scores showed no difference (p = 0.852). In contrast, when scored by the faculty observers, the postintervention score was significantly improved compared to the preintervention score (p < 0.001). In addition, for the faculty observers, the program effect was a significant contributor to the variation in scores. Low intraclass correlation was observed between rater groups.

CONCLUSIONS

This innovative simulation-based program was effective at teaching service recovery communication skills to residents as evaluated by EM faculty, but not PIs. This study supports further exploration into programs to teach and evaluate service recovery communication skills in EM residents.

Effectiveness of a training program on the role recognition and case portrayal of standardized patients

OBJECTIVE

Standardized patient (SP) scenarios could provide medical situations of high fidelity for teaching or examinations, which would otherwise be difficult with real patients. Moreover, SPs can also be used to reliably evaluate or certify the competence of medical trainees. Since the interactions between examinee and SP are dynamic and complex, SPs need to portray the case reliably and consistently across different examinees. Thus, we developed a 1-day program for newly recruited SPs to teach basic competence in case portrayal. The purpose of this study is to assure the effectiveness of this training program in role recognition as a SP and case portrayal.

MATERIALS AND METHODS

A total of 80 SPs were recruited from 2010 to 2013. They were asked to complete questionnaires before and after the training program using a five-point, Likert-type scale (1 - strongly disagree and 5 - strongly agree). The questionnaire comprised 16 items covering SP characteristics, role recognition, and case portrayal. The results were analyzed using PASW Statistics 18, paired t-test, to determine the effectiveness of the training program.

RESULTS

A total of 78 questionnaires were collected for a response rate of 97.5%. The mean age of the SPs was 50.2 ± 7.9 (25-68) years. Most SPs showed improvement in role recognition. In the domain of case portrayal, three of nine items (consistency, obedience, and commitment) showed significant difference (P <0.05).

CONCLUSIONS

This study suggests that this program is useful for reinforcing role recognition in newly recruited SPs. With respect to case portrayal, only consistency, obedience, and commitment were improved.

Physical examination skills training: Faculty staff vs. patient instructor feedback-A controlled trial

BACKGROUND

Standardized patients are widely used in training of medical students, both in teaching and assessment. They also frequently lead complete training sessions delivering physical examination skills without the aid of faculty teaching staff-acting as "patient instructors" (PIs). An important part of this training is their ability to provide detailed structured feedback to students which has a strong impact on their learning success. Yet, to date no study has assessed the quality of physical examination related feedback by PIs. Therefore, we conducted a randomized controlled study comparing feedback of PIs and faculty staff following a physical examination assessed by students and video assessors.

METHODS

14 PIs and 14 different faculty staff physicians both delivered feedback to 40 medical students that had performed a physical examination on the respective PI while the physicians observed the performance. The physical examination was rated by two independent video assessors to provide an objective performance standard (gold standard). Feedback of PI and physicians was content analyzed by two different independent video assessors based on a provided checklist and compared to the performance standard. Feedback of PIs and physicians was also rated by medical students and video assessors using a questionnaire consisting of 12 items.

RESULTS

There was no statistical significant difference concerning overall matching of physician or PI feedback with gold standard ratings by video assessment (p = .219). There was also no statistical difference when focusing only on items that were classified as major key steps (p = .802), mistakes or parts that were left out during physical examination (p = .219) or mistakes in communication items (p = .517). The feedback of physicians was significantly better rated than PI feedback both by students (p = .043) as well as by video assessors (p = .034).

CONCLUSIONS

In summary, our study demonstrates that trained PIs are able to provide feedback of equal quantitative value to that of faculty staff physicians with regard to a physical examination performed on them. However, both the students and the video raters judged the quality of the feedback given by the physicians to be significantly better than that of the PIs.

Psychometric characterization of the obstetric communication assessment tool for medical education: a pilot study

OBJECTIVE

To characterize the psychometric properties of a novel Obstetric Communication Assessment Tool (OCAT) in a pilot study of standardized difficult OB communication scenarios appropriate for undergraduate medical evaluation.

METHODS

We developed and piloted four challenging OB Standardized Patient (SP) scenarios in a sample of twenty-one third year OB/GYN clerkship students: Religious Beliefs (RB), Angry Father (AF), Maternal Smoking (MS), and Intimate Partner Violence (IPV). Five trained Standardized Patient Reviewers (SPRs) independently scored twenty-four randomized video-recorded encounters using the OCAT. Cronbach's alpha and Intraclass Correlation Coefficient-2 (ICC-2) were used to estimate internal consistency (IC) and inter-rater reliability (IRR), respectively. Systematic variation in reviewer scoring was assessed using the Stuart-Maxwell test.

RESULTS

IC was acceptable to excellent with Cronbach's alpha values (and 95% Confidence Intervals [CI]): RB 0.91 (0.86, 0.95), AF 0.76 (0.62, 0.87), MS 0.91 (0.86, 0.95), and IPV 0.94 (0.91, 0.97). IRR was unacceptable to poor with ICC-2 values: RB 0.46 (0.40, 0.53), AF 0.48 (0.41, 0.54), MS 0.52 (0.45, 0.58), and IPV 0.67 (0.61, 0.72). Stuart-Maxwell analysis indicated systematic differences in reviewer stringency.

CONCLUSIONS

Our initial characterization of the OCAT demonstrates important issues in communications assessment. We identify scoring inconsistencies due to differences in SPR rigor that require enhanced training to improve assessment reliability. We outline a rational process for initial communication tool validation that may be useful in undergraduate curriculum development, and acknowledge that rigorous validation of OCAT training and implementation is needed to create a valuable OB communication assessment tool.

The outcome of interprofessional education: Integrating communication studies into a standardized patient experience for advanced practice nursing students

The purpose of this qualitative study was to examine the impact of standardized patient experiences (SPE) in the education of the Advanced Practice Nurse (APN). The education of the APN requires educators to make every attempt to promote competency in the areas of communication and clinical-decision making. SPE programs have been found to improve the interpersonal, problem solving, and critical thinking skills of nursing students. For this research twenty-nine APN students participated in SPEs over the course of two semesters. Fifteen student volunteers of those 29 participants were then interviewed three months after the experience. Results revealed that having an expert in the field of communication studies increased awareness of communication skills and how to improve nurse-patient encounters in the clinical setting. The interprofessional collaboration during the SPEs assisted in facilitating the application of learned communication skills into patient-centered care of the APN student.

Job analysis of standardized patient trainer

PURPOSE

Although standardized patients (SPs) have been used widely in medical education, there has been little research on SP trainers. The purpose of this study was to examine the roles SP trainers by establishing their job description and assessing the importance, frequency, and difficulty of each task element.

METHODS

SP trainers' responsibilities were defined and their job descriptions were developed based on a focus group interview (FGI) of 6 panels. The validity of the duties, tasks, and task elements that were elicited from the FGI was examined by the 6 panels and a medical school professor. The data were collected using a questionnaire survey from 24 SP trainers in April 2011. The questionnaire pertained to the importance, frequency, and difficulty of the duties, tasks, and task elements that were identified in SP trainers' job descriptions.

RESULTS

The job description of SP trainers consisted of 10 duties, 25 tasks, and 76 task elements. The average level of importance of the 10 duties was 4.29; "SP training" was identified as the most important duty (4.79+/-0.41). Of the 76 task elements, "providing training regarding the scoring criteria and standardizing the scoring system" had the highest levels of importance and frequency (4.83+/-0.48, 4.08+/-1.06). "Standardizing the acting abilities of SPs" was the most difficult task element (4.50+/-0.66).

CONCLUSION

By analyzing the job description of SP trainers, this study is expected to help define the roles of SP trainers and facilitate their deployment.