Have standardized patient examinations stood the test of time and experience?

Artificial Intelligence and Healthcare Simulation: The Shifting Landscape of Medical Education

The impact of artificial intelligence (AI) will be felt not only in the arena of patient care and deliverable therapies but will also be uniquely disruptive in medical education and healthcare simulation (HCS), in particular. As HCS is intertwined with computer technology, it offers opportunities for rapid scalability with AI and, therefore, will be the most practical place to test new AI applications. This will ensure the acquisition of AI literacy for graduates from the country's various healthcare professional schools. Artificial intelligence has proven to be a useful adjunct in developing interprofessional education and team and leadership skills assessments. Outcome-driven medical simulation has been extensively used to train students in image-centric disciplines such as radiology, ultrasound, echocardiography, and pathology. Allowing students and trainees in healthcare to first apply diagnostic decision support systems (DDSS) under simulated conditions leads to improved diagnostic accuracy, enhanced communication with patients, safer triage decisions, and improved outcomes from rapid response teams. However, the issue of bias, hallucinations, and the uncertainty of emergent properties may undermine the faith of healthcare professionals as they see AI systems deployed in the clinical setting and participating in diagnostic judgments. Also, the demands of ensuring AI literacy in our healthcare professional curricula will place burdens on simulation assets and faculty to adapt to a rapidly changing technological landscape. Nevertheless, the introduction of AI will place increased emphasis on virtual reality platforms, thereby improving the availability of self-directed learning and making it available 24/7, along with uniquely personalized evaluations and customized coaching. Yet, caution must be exercised concerning AI, especially as society's earlier, delayed, and muted responses to the inherent dangers of social media raise serious questions about whether the American government and its citizenry can anticipate the security and privacy guardrails that need to be in place to protect our healthcare practitioners, medical students, and patients.

"Who Am I and Why Am I Here?" A Scoping Review Exploring the Templates and Protocols That Direct Actors in Their Roles as Simulated (Standardized) Patients

SUMMARY STATEMENT

Simulated patients (SPs) are increasingly used in health education and research. The aim of this article was to investigate templates and protocols that enable SPs to accurately and consistently adopt these roles. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for scoping reviews guided the search strategy for articles that detailed such templates or protocols. Embase Classic + Embase, ProQuest ERIC, Ovid MEDLINE, Ovid EMCare, psycINFO, and Scopus were searched, and 17 articles were included in the review. The templates and protocols that were located differed in structure, length, and depth and were developed or used in medical, nursing, allied health, and veterinary medicine disciplines. The validity, reliability, and replicability of studies are explored, and the quality of reporting is evaluated using the Simulation Research Rubric. Recommendations for increasing the rigor of programs and the reporting of research where SPs are adopted are considered.

Augmenting physician examiner scoring in objective structured clinical examinations: including the standardized patient perspective

In Canada, high stakes objective structured clinical examinations (OSCEs) administered by the Medical Council of Canada have relied exclusively on physician examiners (PEs) for scoring. Prior research has looked at using SPs to replace PEs. This paper reports on two studies that implement and evaluate a standardized patient (SP) scoring tool to augment PE scoring. The unique aspect of this study is that it explores the benefits of combining SP and PE scores. SP focus groups developed rating scales for four dimensions they labelled: Listening, Communication, Empathy/Rapport, and Global Impression. In Study I, 43 SPs from one site of a national PE-scored OSCE rated 60 examinees with the initial SP rating scales. In Study II, 137 SPs used slightly revised rating scales with optional narrative comments to score 275 examinees at two sites. Examinees were blinded to SP scoring and SP ratings did not count. Separate PE and SP scoring was examined using descriptive statistics and correlations. Combinations of SP and PE scoring were assessed using pass-rates, reliability, and decision consistency and accuracy indices. In Study II, SP and PE comments were examined. SPs showed greater variability in their scoring, and rated examinees lower than PEs on common elements, resulting in slightly lower pass rates when combined. There was a moderate tendency for both SPs and PEs to make negative comments for the same examinee but for different reasons. We argue that SPs and PE assess performance from different perspectives, and that combining scores from both augments overall reliability of scores and pass/fail decisions. There is potential to provide examinees with feedback comments from each group.

Medical students' experiences in learning the Mental State Examination with standardized patients

INTRODUCTION

Standardized patients are widely used in psychiatry teaching and examinations, but their effectiveness in teaching psychopathology and the Mental State Examination in particular has not been established. A mixed-methods study was undertaken amongst fourth year medical students during a psychiatry rotation to explore the usefulness of learning the Mental State Examination with standardized patients compared with real patients.

METHODS

Qualitative semistructured interviews (N = 10) were done to explore learning experiences with standardized patients. The Experience with Standardized Patients Scale was used (N = 200) for the quantitative assessment in relation to examination performance and learning experience.

RESULTS

In qualitative interviews, the common themes were that the experience with standardized patients was not natural and made it difficult to suspend disbelief, but students recognized the usefulness of standardized patients for practicing their skills. While there was no significant relationship between overall student experiences with standardized patients and their examination performance in the end-of-rotation Observed Standardized Clinical Examination (P = .40), the exception was with those who perceived they learnt more communication skills with standardized patients (P = .002). Students' experience with standardized patients role-playing psychotic disorders was poorest compared with standardized patients role-playing depressive and neurotic disorders. There were statistically significant differences in experiences between the three diagnostic groups.

DISCUSSION

Standardized patients are useful in psychiatry for students to practice and improve their skills in eliciting psychopathology. However, attention to the training and selection of these individuals for psychiatric roles are crucial.

How do novices learn physical examination skills? A systematic review of the literature

BACKGROUND

Physical Examination (PE) skills are vital for patient care, and many medical students receive their first introduction to them in their pre-clinical years. A substantial amount of curriculum time is devoted to teaching these skills in most schools. Little is known about the best way to introduce PE skills to novice learners.

OBJECTIVE

Our objective was to conduct a systematic review of how medical students are first taught PE skills and the evidence supporting these strategies.

DESIGN

We searched ERIC, SCOPUS, MEDLINE, PubMed and EMBASE for descriptions of complete PE curricula for novice learners. Inclusion criteria were: (1) English language; (2) subjects were enrolled in medical school and were in the preclinical portion of their training; (3) description of a method to teach physical examination skills for the first time; (4) description of the study population; (5) Description of a complete PE curriculum. We used the Medical Education Research Study Quality Instrument (MERSQI) score to evaluate the quality of evidence provided.

RESULTS

Our search returned 5,418 articles; 32 articles met our inclusion criteria. Two main types of curricula were reported: comprehensive 'head-to-toe' PE curricula (18%) and organ system-based curricula (41%). No studies compared these directly, and only two evaluated trainees' clinical performance. The rest of the articles described interventions used across curricula (41%). Median MERSQI score was 10.1 Interquartile range 8.1-12.4. We found evidence for the use of non-faculty teaching associates, technology-enhanced PE education, and the addition of clinical exposure to formal PE teaching.

CONCLUSIONS

The current literature on teaching PE is focused on describing innovations to head-to-toe and organ system-based curricula rather than their relative effectiveness, and is further limited by its reliance on short-term outcomes. The optimal strategy for novice PE instruction remains unknown.

Standardized Patient Encounters: Periodic Versus Postencounter Evaluation of Nontechnical Clinical Performance

INTRODUCTION

Standardized patients are a beneficial component of modern healthcare education and training, but few studies have explored cognitive factors potentially impacting clinical skills assessment during standardized patient encounters. This study examined the impact of a periodic (vs. traditional postencounter) evaluation approach and the appearance of critical verbal and nonverbal behaviors throughout a standardized patient encounter on scoring accuracy in a video-based scenario.

METHODS

Forty-nine standardized patients scored either periodically or at only 1 point in time (postencounter) a healthcare provider's verbal and nonverbal clinical performance during a videotaped standardized patient encounter. The healthcare provider portrayed in this study was actually a standardized patient delivering carefully scripted verbal and nonverbal behaviors in their portrayal of an actual physician. The encounter itself was subdivided into 3 distinct segments for the purpose of supporting periodic evaluation, with the expectation that both verbal and nonverbal cues occurring in the middle segment would be more challenging to accurately report for participants in the postscenario evaluation group as a result of working memory decay.

RESULTS

Periodic evaluators correctly identified a significantly greater number of critical verbal cues midscenario than postencounter evaluators (P < 0.01) and correctly identified a significantly greater number of critical nonverbal cues than their postscenario counterparts across all 3 scenario segments (P < 0.001). Further, postscenario evaluations exhibited a performance decrement in terms of midscenario correct identifications that periodic evaluators did not (P < 0.01). Also, periodic evaluators exhibited fewer verbal cue false-positives during the first segment of the scenario than postscenario evaluators (P < 0.001), but this effect did not extend to other segments regardless of the cue type (ie, verbal or nonverbal).

DISCUSSION

Pausing lengthier standardized patient encounters periodically to allow for more frequent scoring may result in better reporting accuracy for certain clinical behavioral cues. This could enable educators to provide more specific formative feedback to individual learners at the session's conclusion. The most effective encounter design will ultimately depend on the specific goals and training objectives of the exercise itself.

Do Standardized Patients Have Concerns About Students Not Captured by Traditional Assessment Forms?

Construct: Traditionally, standardized patients (SPs) assess students' clinical skills principally through numerical rating forms-an approach that may not fully capture SPs' concerns. SPs are students' closest approximation to real patients. To maximally benefit students' clinical training and evaluation it is important to find ways to give voice to the totality of SPs' perspectives.

BACKGROUND

SPs have been shown to be a reliable and valid means to assess medical students' clinical skills in clinical skills examinations. We noticed, however, that SPs often express "off the record" concerns about students, which they do not include on traditional assessment forms.

APPROACH

To explore these "off the record" concerns, we designed a Concerns item and added it to the traditional assessment form for an end-of-3rd-year clinical skills examination shared by three medical schools. We asked SPs to use this Concerns item to identify students about whom they had any "gut-level" concerns and provided them with a narrative opportunity to explain why. SPs were informed that the purpose of the item was to help students with difficulties and was not part of the student's grade.

RESULTS

We analyzed the concerns data using quantitative and qualitative methods. Of 551 students at three schools, 223 (∼40%) had concerns recorded. Seventy students received two or more concerns. Qualitative analysis of SPs' comments revealed 3 major categories of concern: communication and interpersonal skills, history taking, and physical exam. Grouped under each were several subcategories. More than half of the written comments from the SPs related to the communication/interpersonal skills category and included subcategories commonly addressed in communications courses: lack of empathy, good listening skills, and lack of connection to the patient. They also included subcategories that in our experience are less commonly addressed: odd or off-putting mannerisms, lack of confidence, unprofessional behavior, domineering behavior, and biased behavior. Another 47% of concerns identified deficiencies in history taking and physical examination. Of the students with concerns noted by two or more SPs, SPs' narrative comments on 84%, 42%, and 48% of the students in the domains of communications, history, and physical exam respectively indicated potential problems not identified by scores on the traditional assessment form.

CONCLUSION

The Concerns item is a narrative assessment method that may add value to traditional quantitative scoring by identifying and characterizing problematic student performance not captured by the traditional assessment form. It may thus contribute to giving fuller voice to the totality of SPs' perspective.

Relationship Between Faculty and Standardized Patient Assessment Scores of Podiatric Medical Students During a Standardized Performance Assessment Laboratory

BACKGROUND

Direct assessment of health professional student performance of clinical skills can be accurately performed in the standardized performance assessment laboratory (SPAL), typically by health professional faculty. However, owing to time and economic considerations, nonmedical individuals have been specially trained to perform the same function (standardized patients [SPs]). This study compared the assessment scores of the history and physical examination components of a SPAL designed for second-year podiatric medical students at Des Moines University (DMU) by a podiatry medical faculty member and SPs.

METHODS

A total of 101 students from the classes of 2015 and 2016 were evaluated in 2013 and 2014 by 11 to 13 SPs from the DMU SPAL program. The video recordings of these 101 students were then evaluated by one faculty member from the College of Podiatric Medicine and Surgery at DMU.

RESULTS

The Pearson correlation coefficient for each class showed a strong linear relationship between SP and faculty assessment scores. The associations between SP and faculty assessment scores in the history, physical examination, and combined history and physical examination components for the 2016 class (0.706, 0.925, and 0.911, respectively) were found to be stronger than those for the 2015 class (0.697, 0.791, and 0.791, respectively).

CONCLUSIONS

This study indicated that there are strong associations between the assessment scores of trained SPs and faculty for the history, physical examination, and combined history and physical examination components of second-year SPAL activity for podiatric medical students.

Patient exposure in the basic science classroom enhances differential diagnosis formation and clinical decision-making

Purpose. The authors proposed that introducing real patients into a pathology classroom early in medical education would help integrate fundamental principles and disease pathology with clinical presentation and medical history. Methods. Three patients with different pathologies described their history and presentation without revealing their diagnosis. Students were required to submit a differential diagnosis in writing, and then were able to ask questions to arrive at the correct diagnosis. Students were surveyed on the efficacy of patient-based learning. Results. Average student scores on the differential diagnosis assignments significantly improved 32% during the course. From the survey, 72% of students felt that patient encounters should be included in the pathology course next year. Seventy-four percent felt that the differential diagnosis assignments helped them develop clinical decision-making skills. Seventy-three percent felt that the experience helped them know what questions to ask patients. Eighty-six percent felt that they obtained a better understanding of patients' social and emotional challenges. Discussion. Having students work through the process of differential diagnosis formulation when encountering a real patient and their clinical presentation improved clinical decision-making skills and integrated fundamental concepts with disease pathology during a basic science pathology course.

Accuracy of portrayal by standardized patients: results from four OSCE stations conducted for high stakes examinations

BACKGROUND

The reliability in Objective Structured Clinical Exams (OSCEs) is based on variance introduced due to examiners, stations, items, standardized patients (SP), and the interaction of one or more of these items with the candidates. The impact of SPs on the reliability has not been well studied. Accordingly, the main purpose of the present study was to assess the accuracy of portrayal by standardized patients.

METHODS

Four stations from a ten station high-stakes OSCE were selected for video recording. Due to the large number of candidates to be evaluated, the OSCE was administered using four assessment tracks. Four SPs were trained for each case (n = 16). Two physician assessors were trained to assess the accuracy of SP portrayal using a station-specific instrument based on the station guidelines. For the items with disagreement a third physician was asked to review and the mode was used for analysis. Each instrument included case-specific items on verbal and physical portrayal using a 3-point rating scale ("yes", "yes, but" and "not done"). The physician assessors also scored each SP on their overall performance based on a 5-item anchored global rating scale ("very poor", "poor", "ok", "good", and "very good"). SPs at location 1 were trained by one trainer and SPs at location 2 had another trainer. All SPs were employed in a high-stakes OSCE for at least the second time.

RESULTS

The reliability of rating scores ranged from Cronbach's alpha of .40 to .74. Verbal portrayal by SPs did not significantly differ for most items; however, the facial expressions of the SPs differed significantly (p < .05). An emergency management station that depended heavily on SPs physical presentation and facial expressions differed between all four SPs trained for that station.

CONCLUSIONS

Variation of trained SP portrayal of the same station across different tracks and at different times in OSCE may contribute substantial error to OSCE assessments. The training of SPs should be strengthened and constantly monitored during the exam to ensure that the examinees' scores are a true reflection of their competency and devoid of exam errors.

The Objective Structured Clinical Examination (OSCE): AMEE Guide No. 81. Part II: organisation & administration

The organisation, administration and running of a successful OSCE programme need considerable knowledge, experience and planning. Different teams looking after various aspects of OSCE need to work collaboratively for an effective question bank development, examiner training and standardised patients' training. Quality assurance is an ongoing process taking place throughout the OSCE cycle. In order for the OSCE to generate reliable results it is essential to pay attention to each and every element of quality assurance, as poorly standardised patients, untrained examiners, poor quality questions and inappropriate scoring rubrics each will affect the reliability of the OSCE. The validity will also be influenced if the questions are not realistic and mapped against the learning outcomes of the teaching programme. This part of the Guide addresses all these important issues in order to help the reader setup and quality assure their new or existing OSCE programmes.

Reliability of standardized patients used in a communication study on international nurses in the United States of America

As an evaluation method, standardized patients have a long history in medical education and research yet are less established in nursing. This paper explores the reliability of using standardized patients as the evaluative method in a communication competence pilot study with international nurses. Standardized patients and second raters scored the same encounters. We examined the scores by intraclass correlation coefficients. Anecdotal comments by the two types of raters were assessed qualitatively to highlight similarities and areas of difference between them. The results of reliability analysis for standardized patients scores for the composite variables of Establishing Communicative Rapport, Therapeutic Communication, Non-Verbal Communication, and Overall Satisfaction ranged from 0.755 (P < 0.01) to 0.42 (P = 0.09). In this study, the results showed standardized patient evaluation has moderate to substantial reliability when compared to second raters of the same set of clinical encounters. This is similar to the reliability established over many decades of medical research. Greater use of this dynamic and interactive technique may be beneficial to nursing education and research.

Relationship between standardized patient checklist item accuracy and performing arts experience

INTRODUCTION

: It is not known whether a Standardized Patient's (SP's) performing arts background could affect his or her accuracy in recording candidate performance on a high-stakes clinical skills examination, such as the Comprehensive Osteopathic Medical Licensing Examination Level 2 Performance Evaluation. The purpose of this study is to investigate the differences in recording accuracy of history and physical checklist items between SPs who identify themselves as performing artists and SPs with no performance arts experience.

METHODS

: Forty SPs identified themselves as being performing artists or nonperforming artists. A sample of SP live examination ratings were compared with a second set of ratings obtained after video review (N = 1972 SP encounters) over 40 cases from the 2008-2009 testing cycle. Differences in SP checklist recording accuracy were tested as a function of performing arts experience.

RESULTS

: Mean overall agreement rates, both uncorrected and corrected for chance agreement, were very high (0.94 and 0.79, respectively, at the overall examination level). There was no statistically significant difference between the two groups with respect to any of the mean accuracy measures: history taking (z = -0.422, P = 0.678), physical examination (z = -1.453, P = 0.072), and overall data gathering (z = -0.812, P = 0.417) checklist items.

CONCLUSION

: Results suggest that SPs with or without a performing arts background complete history taking and physical examination checklist items with high levels of precision. Therefore, SPs with and without performing arts experience can be recruited for high-stakes SP-based clinical skills examinations without sacrificing examination integrity or scoring accuracy.

Systems-based practice assessed with a performance-based examination simulated and scored by standardized participants in the health care system: feasibility and psychometric properties

BACKGROUND

Systems-based practice is one of the six general competencies proposed by the Accreditation Council for Graduate Medical Education in their Outcome Project. However, little has been published on its assessment--possibly because the systems-based practice competency has been viewed as difficult to define and measure.

PURPOSE

The purpose of this study was to determine whether a full performance-based examination of systems-based practice cases simulated and scored by standardized participants in the health care system could feasibly be constructed and implemented that would provide reliable and valid measurements.

METHODS

In the 1st year of the project (2008), four systems-based practice cases were developed and pilot tested with 13 residents. Videotapes of residents were studied to develop an instrument for subsequent assessment of performance by standardized participants. In the 2nd year (2009), the examination was expanded to a full 12 cases, which were completed by 11 second-year residents, and psychometric analyses were performed on the scores.

RESULTS

The generalizability coefficient for the full 12-case examination based on scoring by standardized participants was .71, which is nearly equal to that based on scoring by faculty physician observers, which was .78. The correlation between total scores obtained with standardized participants and physician observers was .78.

CONCLUSIONS

A performance-based examination can provide a feasible and reliable assessment of systems-based practice. However, attempts to evaluate convergent validity and discriminant validity-by correlating systems-based practice performance assessments with mean global ratings of residents on the 6 competencies by faculty throughout training-were unsuccessful, due to a lack of independence between the rated dimensions.

How does a comprehensive clinical performance examination relate to ratings on the medical school student performance evaluation?

BACKGROUND

U.S. medical schools have long used the Medical Student Performance Evaluation (MSPE) to represent overall student performance while comprehensive clinical performance exams (CPX) are beginning to emerge as a new standard for determining student competence.

PURPOSE

This study describes the association between the MSPE and CPX in their independent measures of student competence.

METHODS

We examined the relationship between CPX scores and student MSPE rating at our institution, which was completed independently of the CPX.

RESULTS

Students with higher CPX scores had better MSPE rating, but the associations are small ranging from rs=.13 for history-taking skills to rs=.31 for interpersonal skills.

CONCLUSIONS

CPX results are not strongly related to MSPE rating and, thus, they may provide information on clinical competencies that should be included in the MSPE.

Roles of sex and ethnicity in procurement coordinator--family communication during the organ donation discussion

CONTEXT

Interpersonal relations with health care providers influence families' decisions to consent to solid-organ donation. However, previous research has been based on retrospective interviews with donation-eligible families and has not directly examined the interpersonal interactions between families and organ procurement coordinators.

OBJECTIVE

To increase understanding of the interpersonal interaction between procurement coordinators and families during the organ donation discussion, with special attention to the influence of the sex and race of the procurement coordinator and the race of the potential donor's family.

DESIGN

A descriptive study in which standardized patients portrayed family members interacting with actual procurement coordinators in simulated donation request scenarios.

SETTING AND PARTICIPANTS

Thirty-three videotaped interactions between standardized patients and 17 procurement coordinators involving 2 different scenarios depicting deceased donation were evaluated.

MAIN OUTCOME MEASURES

Video recordings were rated by independent coders. Coders completed the Impact Message Inventory-Form C, the Participatory Style of Physician Scale, and the Siminoff Communication and Content and Affect Program-Global Observer Ratings scale.

RESULTS AND CONCLUSIONS

African American procurement coordinators, particularly African American women, were rated as more controlling and work-oriented than white procurement coordinators. Male procurement coordinators were more affiliative with the white family than the African American family, whereas female procurement coordinators were slightly less affiliative with the white family. African American procurement coordinators expressed more positive affect when interacting with the African American family than the white family, whereas the opposite was true for white procurement coordinators. Research is needed to cross-validate these exploratory findings and further examine cultural mistrust between procurement coordinators and families of ethnic minorities, especially given the negative attitudes of many minorities toward donation.

Roles of Sex and Ethnicity in Procurement Coordinator—Family Communication during the Organ Donation Discussion

Context

Interpersonal relations with health care providers influence families' decisions to consent to solid-organ donation. However, previous research has been based on retrospective interviews with donation-eligible families and has not directly examined the interpersonal interactions between families and organ procurement coordinators.

Objective

To increase understanding of the interpersonal interaction between procurement coordinators and families during the organ donation discussion, with special attention to the influence of the sex and race of the procurement coordinator and the race of the potential donor's family.

Design

A descriptive study in which standardized patients portrayed family members interacting with actual procurement coordinators in simulated donation request scenarios.

Setting and Participants

Thirty-three videotaped interactions between standardized patients and 17 procurement coordinators involving 2 different scenarios depicting deceased donation were evaluated.

Main Outcome Measures

Video recordings were rated by independent coders. Coders completed the Impact Message Inventory-Form C, the Participatory Style of Physician Scale, and the Siminoff Communication and Content and Affect Program–Global Observer Ratings scale.

Results and Conclusions

African American procurement coordinators, particularly African American women, were rated as more controlling and work-oriented than white procurement coordinators. Male procurement coordinators were more affiliative with the white family than the African American family, whereas female procurement coordinators were slightly less affiliative with the white family. African American procurement coordinators expressed more positive affect when interacting with the African American family than the white family, whereas the opposite was true for white procurement coordinators. Research is needed to cross-validate these exploratory findings and further examine cultural mistrust between procurement coordinators and families of ethnic minorities, especially given the negative attitudes of many minorities toward donation.

A construct validity study of clinical competence: a multitrait multimethod matrix approach

INTRODUCTION

The purpose of the study was to adduce evidence for estimating the construct validity of clinical competence measured through assessment instruments used for high-stakes examinations.

METHODS

Thirty-nine international physicians (mean age = 41 + 6.5 y) participated in high-stakes examination and 3-month supervised clinical practice to determine the practice readiness of physicians. Three traits-doctor-patient relationship, clinical competence, and communication skills-were assessed with objective structured clinical examinations, in-training evaluation reports, and clinical assessments. These traits were intercorrelated in a multitrait multimethod matrix (MTMM).

RESULTS

The reliability of assessments ranged from moderate to high (Cronbach's alpha: 0.58-0.98; Ep(2) = 0.79). There is evidence for both convergent and divergent validity for clinical competence, followed by doctor-patient relationships, and communications (validity coefficients = 0.12-0.85). The correlations between the same methods but different traits indicate that there is substantial method specificity in the assessment accounting for nearly one-quarter of the variance (23.7%).

DISCUSSION

There is evidence for the construct validity of all 3 traits across 3 methods. The MTMM approach, currently underutilized, could be used to estimate the degree of evidence for validating complex constructs, such as clinical competence.

Evaluation of the FOCUS (Feedback on Counseling Using Simulation) instrument for assessment of client-centered nutrition counseling behaviors

OBJECTIVE

To develop an instrument to assess client-centered counseling behaviors (skills) of student-counselors in a standardized patient (SP) exercise.

METHODS

Descriptive study of the accuracy and utility of a newly developed counseling evaluation instrument. Study participants included 11 female student-counselors at a Midwestern university-10 Caucasian, 1 African-American-for the simulated counseling sessions, in which the Feedback on Counseling Using Simulation (FOCUS) instrument was applied in 2 SP scenarios (cardiovascular disease and diabetes). FOCUS ratings of student-counselors by 4 SPs during 22 sessions were compared with ratings from a 3-member panel of experts who independently viewed the 22 videotaped sessions. Quantitative analysis of instrument validity included inter-rater reliability by computing generalizability coefficients, Pearson correlations, and Spearman rank-order correlations.

RESULTS

FOCUS criteria encompassed relevant dimensions of nutrition counseling based in a client-centered perspective. The critical points of information gathering and counseling behaviors showed internal consistency overall and good inter-rater reliability with the cardiovascular disease scenario. For both scenarios, pooled ratings of 3 experts agreed with ratings carried out by SPs.

CONCLUSIONS AND IMPLICATIONS

Initial findings suggest that the FOCUS instrument with client-centered criteria may enhance evaluation of counseling skills in SP exercises, meriting further study with larger groups.

Trained lay observers can reliably assess medical students' communication skills

CONTEXT

Our project investigated whether trained lay observers can reliably assess the communication skills of medical students by observing their patient encounters in an out-patient clinic.

METHODS

During a paediatrics clerkship, trained lay observers (standardised observers [SOs]) assessed the communication skills of Year 3 medical students while the students interviewed patients. These observers accompanied students into examination rooms in an out-patient clinic and completed a 15-item communication skills checklist during the encounter. The reliability of the communication skills scores was calculated using generalisability analysis. Students rated the experience and the validity of the assessment. The communication skills scores recorded by the SOs in the clinic were correlated with communication skills scores on a paediatrics objective structured clinical examination (OSCE).

RESULTS

Standardised observers accompanied a total of 51 medical students and watched 199 of their encounters with paediatric patients. The reliability of the communication skills scores from nine observed patient encounters was calculated to be 0.80. There was substantial correlation between the communication skills scores awarded by the clinic observers and students' communication skills scores on their OSCE cases (r = 0.53, P < 0.001). Following 83.8% of the encounters, students strongly agreed that the observer had not interfered with their interaction with the patient. After 95.8% of the encounters, students agreed or strongly agreed that the observers' scoring of their communication skills was valid.

CONCLUSIONS

Standardised observers can reliably assess the communication skills of medical students during clinical encounters with patients and are well accepted by students.

A ten-year review of the literature on the use of standardized patients in teaching and learning: 1996-2005

BACKGROUND

Although there is a growing body of literature on the educational use of standardized patients (SP) in teaching and learning, there have been no reviews on their value.

OBJECTIVE

To determine whether the educational use of SPs has an effect on the knowledge, skills, and behaviour of learners in the health professions.

METHODS

English-language articles covering the period 1996-2005 were reviewed to address the issue of to what extent has the use of SPs affected the knowledge, skills and performance of learners. Out of 797 abstracts, 69 articles, which met the review criteria, were selected. An adaptation of Kirkpatrick's model was used to classify and analyse the articles.

RESULTS

Most of the learners were students in medicine and nursing. SPs were used mostly to teach communication skills and clinical skills. The study designs were case-control (29%), pre-test/post-test (24.6%), post-test only (26.1%) and qualitative studies (20.3%). METHODOLOGICAL ISSUES: Most of the studies had weak research designs. More rigorous designs with control or comparison groups should be used in future research.

CONCLUSIONS

Most studies reported that the educational use of SPs was valuable. More rigorous studies would support the evidence-based use of SPs in teaching and learning.

Standardized patients for HIV/AIDS training in resource-poor settings: the expert patient-trainer

This article presents a unique approach to HIV/AIDS training in resource-poor settings that incorporates the use of standardized patients (SPs). Integrated Management of Adolescent and Adult Illness (IMAI) is a World Health Organization health systems strengthening initiative with a strong emphasis on training health workers in the management of common diseases and conditions. In IMAI, SPs are called Expert Patient-Trainers (EPTs) to emphasize their role in the training of health workers. EPTs were first used in IMAI training in Uganda in 2004. Since then, the method has been adopted by a number of other countries in Africa, Latin America, and Asia. EPTs are usually recruited from groups of people living with HIV/AIDS. In the classroom, EPTs discuss living with HIV and help participants understand HIV as it affects patients. Course participants spend approximately two hours per day in "skill stations," multiple-station assessments consisting of one-on-one encounters with EPTs. In each encounter, the health worker interacts with an EPT portraying a standardized case. Instructions on how to portray each case provide only broad outlines of the major clinical and counseling points; the EPT is expected to use his or her own life experiences to fill in emotional details. Course facilitators noted that health workers were often initially skeptical about EPTs, but this generally turned to enthusiasm after participating in the skill stations. EPTs benefited from the sense of being part of the training team, the satisfaction of improving the skills of health workers, and learning more about their illness.

Evaluation of athletic training students' clinical proficiencies

CONTEXT

Appropriate methods for evaluating clinical proficiencies are essential in ensuring entry-level competence.

OBJECTIVE

To investigate the common methods athletic training education programs use to evaluate student performance of clinical proficiencies.

DESIGN

Cross-sectional design.

SETTING

Public and private institutions nationwide.

PATIENTS OR OTHER PARTICIPANTS

All program directors of athletic training education programs accredited by the Commission on Accreditation of Allied Health Education Programs as of January 2006 (n = 337); 201 (59.6%) program directors responded.

DATA COLLECTION AND ANALYSIS

The institutional survey consisted of 11 items regarding institutional and program demographics. The 14-item Methods of Clinical Proficiency Evaluation in Athletic Training survey consisted of respondents' demographic characteristics and Likert-scale items regarding clinical proficiency evaluation methods and barriers, educational content areas, and clinical experience settings. We used analyses of variance and independent t tests to assess differences among athletic training education program characteristics and the barriers, methods, content areas, and settings regarding clinical proficiency evaluation.

RESULTS

Of the 3 methods investigated, simulations (n = 191, 95.0%) were the most prevalent method of clinical proficiency evaluation. An independent-samples t test revealed that more opportunities existed for real-time evaluations in the college or high school athletic training room (t(189) = 2.866, P = .037) than in other settings. Orthopaedic clinical examination and diagnosis (4.37 +/- 0.826) and therapeutic modalities (4.36 +/- 0.738) content areas were scored the highest in sufficient opportunities for real-time clinical proficiency evaluations. An inadequate volume of injuries or conditions (3.99 +/- 1.033) and injury/condition occurrence not coinciding with the clinical proficiency assessment timetable (4.06 +/- 0.995) were barriers to real-time evaluation. One-way analyses of variance revealed no difference between athletic training education program characteristics and the opportunities for and barriers to real-time evaluations among the various clinical experience settings.

CONCLUSIONS

No one primary barrier hindered real-time clinical proficiency evaluation. To determine athletic training students' clinical proficiency for entry-level employment, athletic training education programs must incorporate standardized patients or take a disciplined approach to using simulation for instruction and evaluation.

Validating measures of third year medical students' use of interpreters by standardized patients and faculty observers

BACKGROUND

Increasing prevalence of limited English proficiency patient encounters demands effective use of interpreters. Validated measures for this skill are needed.

OBJECTIVE

We describe the process of creating and validating two new measures for rating student skills for interpreter use.

SETTING

Encounters using standardized patients (SPs) and interpreters within a clinical practice examination (CPX) at one medical school.

MEASUREMENTS

Students were assessed by SPs using the interpreter impact rating scale (IIRS) and the physician patient interaction (PPI) scale. A subset of 23 encounters was assessed by 4 faculty raters using the faculty observer rating scale (FORS). Internal consistency reliability was assessed by Cronbach's coefficient alpha (alpha). Interrater reliability of the FORS was examined by the intraclass correlation coefficient (ICC). The FORS and IIRS were compared and each was correlated with the PPI.

RESULTS

Cronbach's alpha was 0.90 for the 7-item IIRS and 0.88 for the 11-item FORS. ICC among 4 faculty observers had a mean of 0.61 and median of 0.65 (0.20, 0.86). Skill measured by the IIRS did not significantly correlate with FORS but correlated with the PPI.

CONCLUSIONS

We developed two measures with good internal reliability for use by SPs and faculty observers. More research is needed to clarify the reasons for the lack of concordance between these measures and which may be more valid for use as a summative assessment measure.

Impact of student ethnicity and primary childhood language on communication skill assessment in a clinical performance examination

BACKGROUND

Clinical performance examinations (CPX) with standardized patients (SPs) have become a preferred method to assess communication skills in US medical schools. Little is known about how trainees' backgrounds impact CPX performance.

OBJECTIVE

The objective of this paper is to examine the impact of student ethnicity, primary childhood language, and experience of diversity on the communication scores of a high-stakes CPX using SPs.

DESIGN

This research was designed as an observational study.

PARTICIPANTS

The participants of this study were third-year medical students at one US medical school.

MEASUREMENTS AND MAIN RESULTS

The measurements used in this study were CPX scores from mandatory exam, student demographics and experience with diversity measured by self-report on a survey, and Medical College Admission Test (MCAT) and United States Medical Licensing Examination (USMLE) scores. A total of 135 students participated. Asian and black students scored lower than white students on the communication portion of the CPX by approximately half a standard deviation (Asian, 67.4%; black, 64.4%; white, 69.4%, p < .05). There were no differences by ethnicity on history/physical exam scores. Multivariate analysis controlling for MCAT verbal scores reduced ethnic differences in communication scores (Asian-white mean differences = 1.95, p = 0.02), but Asian-white differences were eliminated only after sequential models included primary childhood language (difference = 0.57, p = 0.6).

CONCLUSIONS

Even after controlling for English language knowledge as measured in MCAT verbal scores, speaking a primary childhood language other than English is associated with lower CPX communication scores for Asian students. While poorer communication skills cannot be ruled out, SP exams may contain measurement bias associated with differences in childhood language or culture. Caution is indicated when interpreting CPX communication scores among diverse examinees.

Can patient-centered attitudes reduce racial and ethnic disparities in care?

PURPOSE

Patient-centeredness has been advocated to reduce racial/ethnic disparities in health care quality, but no empirical data support such a connection. The authors' purpose was to determine whether students with patient-centered attitudes have better performance and are less likely to demonstrate disparities with African American compared with white standardized patients (SPs).

METHOD

Third-year medical students were assessed by SPs at the Clinical Educational Center of the Johns Hopkins University School of Medicine in 2002. One African American and one white actor were trained as SPs for each of four case scenarios; students were randomly assigned to interact with either SP for each case. Before the exam, students were surveyed about their attitudes towards patient-centered medicine. Students with and without patient-centered attitudes were compared with regard to their performance with African American and white SPs. Outcome measures were student exam scores in interpersonal skill, history taking, physical exam, and counseling.

RESULTS

All 177 of eligible students participated in all four case scenarios. With white SPs, students with patient-centered attitudes performed similarly to students without patient-centered attitudes in all four areas. However, with African American SPs, students with patient-centered attitudes performed significantly better than students without patient-centered attitudes in interpersonal skills (71.4 versus 69.4, P = .010), history taking (63.8 versus 61.1, P = .003), and counseling (92.1 versus 88.7, P = .002) and not significantly different in physical exam performance (73.6 versus 68.6, P = .311).

CONCLUSIONS

Patient-centered attitudes may be more important in improving physician behaviors with African American patients than with white patients and may, therefore, play a role in reducing disparities.

Improving medical student intensive care unit communication skills: a novel educational initiative using standardized family members

OBJECTIVE

To determine whether intensive care unit (ICU) communication skills of fourth-year medical students could be improved by an educational intervention using a standardized family member.

DESIGN

Prospective study conducted from August 2003 to May 2004.

SETTING

Tertiary care university teaching hospital.

PARTICIPANTS

All fourth-year students were eligible to participate during their mandatory four-week critical care medicine clerkship.

INTERVENTIONS

The educational intervention focused on the initial meeting with the family member of an ICU patient and included formal teaching of a communication framework followed by a practice session with an actor playing the role of a standardized family member of a fictional patient. At the beginning of the critical care medicine rotation, the intervention group received the educational session, whereas students in the control group did not.

MEASUREMENTS AND MAIN RESULTS

At the end of each critical care medicine rotation, all students interacted with a different standardized family member portraying a different fictional scenario. Sessions were videotaped and were scored by an investigator blinded to treatment assignment using a standardized grading tool across four domains: a) introduction; b) gathering information; c) imparting information; and d) setting goals and expectations. A total of 106 (97% of eligible) medical students agreed to participate in the study. The total mean score as well as the scores for the gathering information, imparting information, setting goals, and expectations domains for the intervention group were significantly higher than for the control group (p < .01).

CONCLUSIONS

The communication skills of fourth-year medical students can be improved by teaching and then practicing a framework for an initial ICU communication episode with a standardized family member.

Using infant mannequins in objective standardized clinical examinations: are there unintended consequences?

OBJECTIVE

In objective standardized clinical examination (OSCE) of infants, real infants are generally not used. Instead, the standardized patient portrays a parent who answers a student's questions, and there is no physical examination. One way to assess physical examination skills in these encounters is to have students demonstrate the appropriate examination on a mannequin. But before using this approach, we wanted to assess whether having students examine mannequins affects their history-collecting or communication skills.

METHODS

Third-year medical students were randomized to 2 versions of an infant OSCE case. During the encounter, controls were handed a printed listing of all physical examination (PE) findings. Students in the mannequin group were told relevant PE findings only after the students had examined a part of the mannequin. Student performances on the OSCE case and perceptions about the case were compared.

RESULTS

Thirty-two students were in the control group, and 35 students examined mannequins. No differences were found in total case score (P = .78), or on history-gathering skills (P = .86) and communication skills subscales (P = .78). In addition, questionnaires completed by students after the encounter indicated that the infant mannequins did not affect student perceptions about the realism of the case (P = .91).

CONCLUSIONS

Student performances at collecting the clinical history or communicating with the standardized patient were not adversely affected by inclusion of an infant mannequin. This suggests mannequins can be used to assess students' knowledge of the relevant case-specific PE without adverse effect.

An interprofessional activity using standardized patients

OBJECTIVES

To describe the development and implementation of an interprofessional activity using standardized patients.

METHODS

In the interprofessional standardized patient experience (ISPE), pharmacy students are teamed with medical and nursing students. This team completes an assessment of a standardized patient. During this assessment, each student has time to interview the patient according to his/her own skills and patient care perspective. After the assessment is completed, the team collaborates to develop a patient care plan. Pre-experience and post-experience surveys were conducted.

RESULTS

Pharmacy students' appreciation for each profession's role in patient care increased.

CONCLUSIONS

Despite multiple challenges, an interprofessional activity involving multiple health professionals from multiple institutions can be successfully implemented. Feedback from pharmacy students indicated that this activity increased students' awareness and ability to work as members of the health professional team.

Decision support for acute problems: the role of the standardized patient in usability testing

For applications that require clinician use while interacting with patients, usability testing with standardized patients has the potential to approximate actual patient care in a controlled setting. We used hypothetical scenarios and a standardized patient to collect quantitative and qualitative results in testing an early prototype of a new application, the Acute Respiratory Infection (ARI) Smart Form. The standardized patient fit well into the usability testing sessions. Clinicians had a positive response to the standardized patients and behaved as they normally would during a clinical encounter. Positive findings of the ARI Smart Form included that users thought it had impressive functionality and the potential to save time. Criticism focused on the visual design, which could be streamlined, and navigation, which was difficult in some areas. Based on these results, we are modifying the ARI Smart Form in preparation for use in actual patient care. Standardized patients should be considered for usability testing, especially if an application is to be used during the patient interview.

Physician responses to ambiguous patient symptoms

OBJECTIVE

To examine how primary care physicians respond to ambiguous patient symptom presentations.

DESIGN

Observational study, using thematic analysis within a larger cross-sectional study employing standardized patients (SPs), to describe physician responses to ambiguous patient symptoms and patterns of physician-patient interaction.

SETTING

Community-based primary care offices within a metropolitan area.

PARTICIPANTS

Twenty-three primary care physicians (internists and family physicians).

METHOD

Participating physicians had 2 unannounced SP visits randomly inserted into their daily practice schedules and the visits were audiotaped and transcribed. A coding system focusing on physician responses to concerned patients presenting with ambiguous symptoms was developed through an inductive process. Thematic analyses were then applied to coded data.

RESULTS

Physicians' responses to ambiguous symptoms were categorized into 2 primary patterns: high partnering (HP) and usual care (UC). HP was characterized by greater responsiveness to patients' expression of concern, positivity, sensitivity to patients' clues about life circumstances, greater acknowledgment of symptom ambiguity, and solicitation of patients' perspectives on their problems. UC was characterized by denial of ambiguity and less inclusion of patients' perspectives on their symptoms. Neither HP physicians nor UC physicians actively included patients in treatment planning.

CONCLUSIONS

Primary care physicians respond to ambiguity by either ignoring the ambiguity and becoming more directive (UC) or, less often, by acknowledging the ambiguity and attempting to explore symptoms and patient concerns in more detail (HP). Future areas of study could address whether physicians can learn HP behaviors and whether HP behaviors positively affect health outcomes.