Evaluating communication skills in the OSCE format: reliability and generalizability

An Applied Linguistics Study of How Students Prevent Embarrassments and Impositions During Interactive Examination OSCEs

OBJECTIVE

To assess the effectiveness of politeness strategies used by pharmacy students to avoid embarrassing or imposing on others during objective structured clinical examinations.

METHODS

A total of 19 objective structured clinical examinations video recordings of 10 students (participants) interacting with mock patients were examined using the framework of Politeness Theory (PT). All relevant participant acts (speech activities) were coded using PT into (1) type of face threatening acts (ie, potentially sensitive situations-as regarded by PT) and (2) politeness strategies used to mitigate them. Conversation Analysis was then used to examine the effectiveness of conversational strategies by judging the 'patient' responses to these strategies.

RESULTS

Most acts had the potential to impact patients' negative face needs (ie, desire to act autonomously, eg, upon the practitioner making a request), positive face needs (ie, desire to be liked, eg, upon the practitioner making a diagnosis), or both. Despite applying a variety of positive politeness strategies (eg, avoiding disagreement, or expressing understanding) to prevent embarrassment to the patient, and negative politeness strategies (eg, being indirect, using hedging, or minimizing the imposition) to avoid directly imposing on them, "dispreferred responses" showed participants mostly focused on avoiding impositions, corresponding to what they have been taught, rather embarrassments.

CONCLUSION

Participants were less aware that discussing sensitive topics could cause embarrassment to patients, with the potential to upset them. Developing teaching and evaluation methods to consider patients' face needs could help in assessing and improving pharmacy students' communication skills.

Evaluating the Construct Validity of Competencies: A Retrospective Analysis

Background

A competency-based framework focuses on alignment between professional standards and assessment design. This alignment implies improved measurement validity, yet it has not been established that competence in one context predicts performance in another context. High-stakes competence assessments offer insights into the relationship between assessment design and competencies.

Methods/Analyses

The internationally educated nurses competency assessment program (IENCAP) was developed at Touchstone Institute in collaboration with the College of Nurses of Ontario (CNO) and includes a 12-station OSCE. Each station evaluated the same 10 competencies. We submitted competency scores to a multi-trait multi-method matrix analysis to evaluate the convergent and discriminant validity of competencies.

Results/Observations

All correlations were significant and positive; however, we did not find evidence of convergent or discriminant validity. Correlations were higher between different competencies evaluated within the same station (mean correlation = 0.60) compared to identical competencies evaluated across different stations (mean correlation = 0.19).

Discussion

The results do not provide evidence of construct validity for competencies. While competency-based approaches emphasize various generalized knowledge, skills, and attitudes, these findings indicate that the clinical context is a major determinant of performance.

Conclusion

The context-dependent nature of competencies requires multiple assessments in varied contexts. Performance on a single competency cannot be determined in a single occasion.

Supplementary Information

The online version contains supplementary material available at 10.1007/s40670-023-01794-z.

Best practices for teaching psychosomatic obstetrics and gynecology

Psychosomatic obstetrics and gynecology (POG) encompasses a wide variety of topics. While specialists in this field agree that it is important for practicing clinicians to be familiar with psychosomatic issues related to women's health, there is no consensus about the best practices for teaching and assessing this knowledge, or even which are the topics that should be included. By examining existing literature on educational methodology, this paper aims to suggest best practices that are proven useful in teaching issues related to POG. The paper considers learning objectives for what should be taught, recommendations as to best practices for curriculum delivery, and suggestions for how to assess learners' ability to meet the objectives. Establishing guidelines for teaching POG may be useful for learners at different levels and in various settings.

What do we know about written assessment of health professionals' communication skills? A scoping review

OBJECTIVE

The aim of this scoping review was to investigate the published literature on written assessment of communication skills in health professionals' education.

METHODS

Pubmed, Embase, Cinahl and Psychnfo were screened for the period 1/1995-7/2020. Selection was conducted by four pairs of reviewers. Four reviewers extracted and analyzed the data regarding study, instrument, item, and psychometric characteristics.

RESULTS

From 20,456 assessed abstracts, 74 articles were included which described 70 different instruments. Two thirds of the studies used written assessment to measure training effects, the others focused on the development/validation of the instrument. Instruments were usually developed by the authors, often with little mention of the test development criteria. The type of knowledge assessed was rarely specified. Most instruments included clinical vignettes. Instrument properties and psychometric characteristics were seldom reported.

CONCLUSION

There are a number of written assessments available in the literature. However, the reporting of the development and psychometric properties of these instruments is often incomplete. Practice implications written assessment of communication skills is widely used in health professions education. Improvement in the reporting of instrument development, items and psychometrics may help communication skills teachers better identify when, how and for whom written assessment of communication should be used.

Evaluating the Global Rating scale's psychometric properties to assess communication skills of undergraduate medical students in video-recorded simulated patient encounters

OBJECTIVES

To evaluate the psychometric properties of the Global Rating scale (GR) as an observer-based tool to assess communication skills of undergraduate medical students in video-recorded patient encounters.

METHODS

Seventy advanced undergraduate medical students participated in a simulation-based assessment including patient consultations. Simulated patients rated these encounters with the Consultation and Relational Empathy (CARE) scale. Two independent, blinded raters assessed the videos of the encounters with the GR and another blinded rater with the Clinical Reasoning Indicators Scale (CRI-HT-S). To assess the GR's psychometric properties, we analysed reliability by means of a G-study, interrater reliability by ICC, convergent validity (correlation of GR and CARE), and divergent validity (correlation of GR and CRI-HT-S).

RESULTS

We analysed 325 videos of 65 students (56.9% female, mean age 26.1 ± 2.2 years). The G-coefficient was.90. Interrater reliability of the GR was ICC = .95, 95% CI [.91,.97]. CARE and GR correlated moderately (ρ = .47, 95% CI [.25,.65]). GR and CRI-HT-S did not correlate (ρ = .09, 95% CI [-.16,.34]).

CONCLUSIONS

With excellent reliability and adequate validity, the quality of the GR as assessment instrument for communication skills could be demonstrated.

PRACTICE IMPLICATIONS

The GR is a suitable instrument for video-based rating of communication skills.

Validity Evidence Supporting the Comunicación y Habilidades Interpersonales (CAI) Scale for Medical Spanish Communication and Interpersonal Skills Assessment

PURPOSE

While many schools teach medical Spanish to improve communication with the growing Spanish-speaking population, a standardized assessment methodology is lacking. This study evaluates validity evidence for a newly developed Spanish Communication and Interpersonal Skills (CIS) scale.

METHOD

The authors developed the Comunicación y Habilidades Interpersonales (CAI, pronounced /kī/) scale by culturally and linguistically adapting a 14-item validated English CIS scale. Sources of validity evidence were gathered targeting content, response process, internal structure, relations to other variables, and consequences. The CAI was administered as part of an objective structured clinical examination at 2 medical schools from April to June 2020.

RESULTS

The authors mapped CAI items onto USMLE Step 2 CIS behavioral functions and medical Spanish learner competencies to examine content validity. The mean item difficulty and item discrimination estimates are 2.86 (SD = 0.34) and 0.81 (SD = 0.08), respectively, demonstrating good psychometric properties at the item level. Internal-consistency reliability for a single case was 0.96. Learner variance accounted for 12% of total variance, demonstrating ability to differentiate higher and lower performing learners; the person-case interaction accounted for 44% of total variance, indicating strong case specificity.

CONCLUSIONS

The CAI has higher case specificity than previously reported in English, suggesting that the nuanced contextual elements of individual cases may matter even more when learners are using limited language skills. Implementing the CAI more widely may promote inclusion of patient-centered communication skills as part of curricular content and activities. This study's validity evidence may inform the development of a future medical Spanish certification examination.

Construct Validation of the American Board of Anesthesiology's APPLIED Examination for Initial Certification

BACKGROUND

The American Board of Anesthesiology administers the APPLIED Examination as a part of initial certification, which as of 2018 includes 2 components-the Standardized Oral Examination (SOE) and the Objective Structured Clinical Examination (OSCE). The goal of this study is to investigate the measurement construct(s) of the APPLIED Examination to assess whether the SOE and the OSCE measure distinct constructs (ie, factors).

METHODS

Exploratory item factor analysis of candidates' performance ratings was used to determine the number of constructs, and confirmatory item factor analysis to estimate factor loadings within each construct and correlation(s) between the constructs.

RESULTS

In exploratory item factor analysis, the log-likelihood ratio test and Akaike information criterion index favored the 3-factor model, with factors reflecting the SOE, OSCE Communication and Professionalism, and OSCE Technical Skills. The Bayesian information criterion index favored the 2-factor model, with factors reflecting the SOE and the OSCE. In confirmatory item factor analysis, both models suggest moderate correlation between the SOE factor and the OSCE factor; the correlation was 0.49 (95% confidence interval [CI], 0.42-0.55) for the 3-factor model and 0.61 (95% CI, 0.54-0.64) for the 2-factor model. The factor loadings were lower for Technical Skills stations of the OSCE (ranging from 0.11 to 0.25) compared with those of the SOE and Communication and Professionalism stations of the OSCE (ranging from 0.36 to 0.50).

CONCLUSIONS

The analyses provide evidence that the SOE and the OSCE measure distinct constructs, supporting the rationale for administering both components of the APPLIED Examination for initial certification in anesthesiology.

Are different medical school admission tests associated with the outcomes of a simulation-based OSCE?

BACKGROUND

Medical school admission procedures have the common goal to select applicants with the greatest potential of becoming successful physicians. Hamburg Medical Faculty selects medical students by grade point average (GPA) and employs a two-step selection process of a natural sciences test (HAM-Nat), in some cases followed by multiple mini-interviews (HAM-Int). Multiple mini-interviews can predict non-cognitive outcomes, while GPA has predictive validity for cognitive outcomes. The aim of our study was to explore communication skills and clinical knowledge of advanced medical students according to their respective admission procedure.

METHODS

In July 2019, 146 students grouped according to their admission procedure into GPA-only (19.2 %), HAM-Nat (33.6 %), HAM-Int (30.8 %), and Waiting List (16.4 %) participated in four OSCE stations which equally assessed students' communication skills (OSCE part 1) and clinical knowledge (OSCE part 2) in simulated patient encounters, rated by physicians with checklists. Additionally, psychosocial assessors ranked communication skills with a global rating scale (GR). The students also participated in a multiple choice (MC) exam testing clinical knowledge. Kruskal-Wallis analyses of variance of test performance and Spearman correlation of instruments were calculated.

RESULTS

Students from the Waiting List group performed significantly worse on the MC exam compared to GPA-only and HAM-Int (adjusted p = .029 and 0.018, respectively). No significant differences were found between the admission groups with respect to communication skills. Global Rating and OSCE part 1 (communication) correlated significantly (ρ = 0.228, p = .006) as did OSCE part 2 (clinical knowledge) and MC exam (ρ = 0.242, p = .003), indicating criterion validity. Constructs did not overlap, indicating divergent validity.

CONCLUSIONS

Advanced medical students selected for undergraduate studies by multiple mini-interviews assessing psychosocial skills showed similar communication skills compared to students admitted to medical school by other entryways. It is unclear whether these similarities are due to an effective undergraduate longitudinal communication curriculum. Assessing baseline communication skills of all medical students at entry-level may aid with this question.

Simulated patient and role play methodologies for communication skills and empathy training of undergraduate medical students

BACKGROUND

Verbal and non-verbal communication, as well as empathy are central to patient-doctor interactions and have been associated with patients' satisfaction. Non-verbal communication tends to override verbal messages. The aim of this study was to analyze how medical students use verbal and non-verbal communication using two different educational approaches, student role play (SRP) and actor simulated patient (ASP), and whether the non-verbal behaviour is different in the two different poses.

METHODS

Three raters evaluated 20 students playing the doctor role, 10 in the SRP group and 10 in the ASP group. The videos were analyzed with the Calgary-Cambridge Referenced Observation Guide (CCG) and, for a more accurate evaluation of non-verbal communication, we also evaluated signs of nervousness, and posture. Empathy was rated with the CARE questionnaire. Independent Mann Whitney U tests and Qhi square tests were performed for statistical analysis.

RESULTS

From the 6 main tasks of the CCG score, we obtained higher scores in the ASP group for the task 'Gathering information' (p = 0.0008). Concerning the 17 descriptors of the CCG, the ASP group obtained significantly better scores for 'Exploration of the patients' problems to discover the biomedical perspective' (p = 0.007), 'Exploration of the patients' problems to discover background information and context' (p = 0.0004) and for 'Closing the session - Forward planning' (p = 0.02). With respect to non-verbal behaviour items, nervousness was significantly higher in the ASP group compared to the SRP group (p < 0.0001). Concerning empathy, no differences were found between the SRP and ASP groups.

CONCLUSIONS

Medical students displayed differentiated verbal and non-verbal communication behaviour during the two communication skills training methodologies. These results show that both methodologies have certain advantages and that more explicit non-verbal communication training might be necessary in order to raise students' awareness for this type of communication and increase doctor-patient interaction effectiveness.

Improving the assessment of communication competencies in a national licensing OSCE: lessons learned from an experts' symposium

BACKGROUND

As the communication competencies of physicians are crucial for providing optimal patient care, their assessment in the context of the high-stakes Objective Structured Clinical Examination (OSCE) is of paramount importance. Despite abundant literature on the topic, evidence-based recommendations for the assessment of communication competencies in high stakes OSCEs are scarce. As part of a national project to improve communication-competencies assessments in the Swiss licensing exam, we held a symposium with national and international experts to derive corresponding guidelines.

METHODS

Experts were invited on account of their recognized expertise either in teaching or assessing communication competencies, or in conducting national high-stakes OSCEs. They were asked to propose concrete solutions related to four potential areas for improvement: the station design, the rating tool, the raters' training, and the role of standardized patients. Data gene.rated in the symposium was available for analysis and consisted of video recordings of plenary sessions, of the written summaries of group work, and the cards with participants' personal take-home messages. Data were analyzed using a thematic analysis approach.

RESULTS

Nine major suggestions for improving communication-competencies assessments emerged from the analysis and were classified into four categories, namely, the roles of the OSCE scenarios, rating tool, raters' training, and simulated patients.

CONCLUSION

In the absence of established evidence-based guidelines, an experts' symposium facilitated the identification of nine practical suggestions for improving the assessment of communication competencies in the context of high-stakes OSCEs. Further research is needed to test effectiveness of the suggestions and how they contribute to improvements in the quality of high-stakes communication-competencies assessment.

Working experience is not a predictor of good communication: Results from a controlled trial with simulated patients

INTRODUCTION

The importance of good communication in dentistry is proven in terms of both medical satisfaction and patient-related recovery and prevention. The present work deals with the comparison of communicative abilities and the communicative self-assessment of licensed dentists as well as students of dentistry with special emphasis on the influence of the treatment experience.

MATERIALS AND METHODS

A total of 34 dentists (experimental group) with an average work experience of 16 years and 36 students (control group) with an average of 1.4 years of treatment experience were included. In addition to a tutor, four types of simulation patients with standardised trained roles (anxious, critical, dissatisfied and difficult to motivate) were used to create reproducible conversations. The self-assessment and evaluation of the conversation took place by completing questionnaires. Here, an introductory questionnaire was distributed to the participants prior to the conversation and another one after intervention. Whilst the tutors completed their survey during the intervention, the simulation patients answered their questions after the conversation.

RESULTS

The results showed that the dentists rated their own communication skills significantly higher than the students for anxious (P < 0.001) and unmotivated patients (P = 0.026). However, the evaluation of the simulated patients showed that the students achieved higher overall empathy scores (42.03 vs 38.77, P = 0.016).

CONCLUSION

Due to the declining empathy values with increasing treatment experience, communication training is useful for the daily routine of treatment even for experienced dentists.

Supervision of Resident Physicians

Supervision of resident physicians is a high-risk area of emergency medicine, and what constitutes appropriate supervision is a complex question. In this article, policies and procedures for appropriate supervision of resident physicians and the implications for billing are reviewed. Recommendations on supervision of resident physicians in the emergency department are detailed, with attention paid to addressing challenges in balancing patient safety with resident autonomy and education during the course of patient care and graduate medical education.

Assessing the growth in clinical skills using a progress clinical skills examination

BACKGROUND

This study evaluates the generalizability of an eight-station progress clinical skills examination and assesses the growth in performance for six clinical skills domains among first- and second-year medical students over four time points during the academic year.

METHODS

We conducted a generalizability study for longitudinal and cross-sectional comparisons and assessed growth in six clinical skill domains via repeated measures ANOVA over the first and second year of medical school.

RESULTS

The generalizability of the examination domain scores was low but consistent with previous studies of data gathering and communication skills. Variations in case difficulty across administrations of the examination made it difficult to assess longitudinal growth. It was possible to compare students at different training levels and the interaction of training level and growth. Second-year students outperformed first-year students, but first-year students' clinical skills performance grew faster than second-year students narrowing the gap in clinical skills over the students' first year of medical school.

CONCLUSIONS

Case specificity limits the ability to assess longitudinal growth in clinical skills through progress testing. Providing students with early clinical skills training and authentic clinical experiences appears to result in the rapid growth of clinical skills during the first year of medical school.

Simulated patient's feedback to improve communication skills of clerkship students

BACKGROUND

The changing trends of the society and revisions to medical education have changed the way medical students are trained to adroitly care for patients hence, patient centered care has become need of today's society and communication skills are imperative in developing patient physician relationship. Increasingly, simulations are being used to aid medical students to incorporate theoretical knowledge into practice. There are innumerable studies regarding communication skills in terms of reliability, validity and feasibility but no such study has been documented using simulated patient's feedback in improving communication skills in Pakistan. The aim of this study is to explore whether simulated patients' feedback improves the communication skills of undergraduate medical students.

METHODS

During a randomized control trail a group of eighty students in the final year clerkship at Al-Nafees Medical College have participated in pre-post Objective Structured Clinical Exam (OSCE) on communication skills. The students were selected through convenience sampling technique. Four Objective Structured Clinical Exam (OSCE) stations based on different scenarios of communication skills were developed. Each station of fifteen minutes duration was assessed by both simulated patients and faculty using a validated tool LCSAS (Liverpool Communication Skills Assessment Scale). The difference between the pre and post-tests of two groups was explored by applying independent t-test. Cronbach's alpha was used to check the reliability of scores and effect size was calculated.

RESULTS

Results of this study have showed that there is significant improvement in communication skills after receiving feedback from simulated patients (p value ≤0.05) was observed. An overall Cronbach α = 0.83 on LCSAS reveal a high internal consistency and there was adequate demonstration of effect size(r = 0.8).

CONCLUSION

The results on the scores of the students on the Liverpool Communication Skills Assessment Scale confirm that simulated patient's feedback is essential to enhance the communication skills of the medical students. This study offers significant evidence towards successful conduction of a formal communication skills development initiative at Al-Nafees Medical College using simulated patient feedback during teaching and assessments.

Development of an empathy and clarity rating scale to measure the effect of medical improv on end-of-first-year OCSE performance: a pilot study

Patients want empathetic physicians who listen and understand. How do you teach and measure empathy? Medical educators, including those inspired by Alan Alda, have turned to theater to teach skills in empathetic communication. Improvisation-informedcurriculum (medical improv) draws upon foundational actors training: deep listening, emotional understanding, connections, authenticity. Arating scale to measure the impact of medical improv on empathetic and clear communication does not exist. Objective: To develop aframework and instrument, the Empathy and Clarity Rating Scale (ECRS), for measuring communication elements used by actors and physicians, and pilot ECRS to test effectiveness of medical improv on first-yearstudents' communication skills. Design: Four medical schools collaborated. USMLE Step 2 Communication and Interpersonal Skills (CIS) domains were used as framework for discussion among three focus groups, each with clinicians, actors, communication experts, and community members with patient experience. Audiotaped discussions were transcribed; open coding procedures located emerging themes. The initial coding scheme was compared with the Consultation and Relational Empathy (CARE) measure. ECRS content was aligned with CARE, CIS and focus group themes. Modified nominal processes were conducted to finalize the scale. We implemented procedures to establish content validity and interrater reliability. Final ECRS was used to study student performance across three levels of experience with medical improv. Results: The final ECRS was comprised of seven five-pointscale items. Narrative comments precede behaviorally anchored ratings: 5=desired, 1=ineffective, 2-4=developing based upon adjustment needed. Rater agreement across all items was 84%. There was asmall correlation between the ECRS and another measure interviewing (r=0.262, p=0.003). Students with advanced medical improv training outperformed those without (F=3.51, p=.042). Conclusion: Acommunication scale enlightened by experiences of actors, clinicians, scholars and patients has been developed. The ECRS has potential to detect the impact of medical improv on development of empathetic and clear communication.

Reliability of narrative assessment data on communication skills in a summative OSCE

OBJECTIVE

To quantitatively estimate the reliability of narrative assessment data regarding student communication skills obtained from a summative OSCE and to compare reliability to that of communication scores obtained from direct observation.

METHODS

Narrative comments and communication scores (scale 1-5) were obtained for 14 graduating pharmacy students across 6 summative OSCE stations with 2 assessors per station who directly observed student performance. Two assessors who had not observed the OSCE reviewed narratives and independently scored communication skills according to the same 5-point scale. Generalizability theory was used to estimate reliability. Correlation was used to evaluate the relationship between scores from each assessment method.

RESULTS

A total of 168 narratives and communication scores were obtained. The G-coefficients were 0.571 for scores provided by assessors present during the OSCE and 0.612 for scores from assessors who provided scores based on narratives only. Correlation between the two sets of scores was 0.5.

CONCLUSION

Reliability of communication scores is not dependent on whether assessors directly observe student performance or assess written narratives, yet both conditions appear to measure communication skills somewhat differently.

PRACTICE IMPLICATIONS

Narratives may be useful for summative decision-making and help overcome the current limitations of using solely quantitative scores.

Communication Course for Pediatric Providers Improves Self-efficacy

BACKGROUND

Communication is essential to building a trusting, clinician-patient relationship. Multiple studies have demonstrated the effects of experiential communication training on patient experience and provider well-being and resiliency. To date, no studies have described an organization-wide communication training program for pediatric clinicians. The objective of this study was to evaluate the impact of a pediatric-focused communication course on provider satisfaction, self-efficacy, and burnout.

METHODS

Texas Children's Hospital, in collaboration with the Academy on Communication in Healthcare, designed and implemented a pediatric focused communication course entitled Breakthrough Communication. Pre, immediate-post, and 3-month postcourse completion online surveys were sent to participants 1 day before, 1 day after, and 3 months after course completion. Participant demographic information, self-assessment of communication skills, the Maslach Burnout Inventory Human Services Survey, and postcourse satisfaction data were collected.

RESULTS

Participants reported high course satisfaction and improved self-efficacy in all measured skill sets both following and 3 months after course completion. Trends indicating a reduction in provider burnout improved in 2 of the 3 Maslach Burnout Inventory domains; however, statistical significance was not achieved.

CONCLUSIONS

A pediatric-focused communication course was well received by multi-specialty clinicians within a large, academic health care organization. This course enhanced clinician self-efficacy with newly learned pediatric encounter specific communication skills.

"Hot Seat" Simulation Model for Conflict Resolution: A Pilot Study

BACKGROUND

Unresolved conflicts in health care threaten both clinician morale and quality of patient care. We piloted a training model that targeted clinicians' conflict resolution skills.

METHODS

Sixty clinicians from local hospitals were randomized into an intervention group (n = 30), completing a 3-hour conflict resolution training session, and a control group (n = 30) without training. The training included facilitated practice with actors, coaching, and feedback. Evaluation of 60 participants' conflict resolution skills was done in videotaped simulations with actors portraying interprofessional colleagues. Global ratings and checklist items developed for assessing clinicians' performance mirrored steps in the conflict communication model.

RESULTS

The intervention group's performance exceeded the control group on global scores, 7.2 of 10 (SD = 1.6) versus 5.6 (SD = 1.5), p < .05, and checklist scores, 9.3 of 11 (SD = 2.9) versus 7.9 (SD = 1.5), p < .05. Two checklist items showed statistically significant differences: (1) subjects opened the dialogue on a neutral ground before jumping into conflict discussions (intervention: 97% and control: 73%, p < .05) and (2) subjects elicited the colleague's story before sharing their own story (intervention: 70% and control: 27%, p < .05).

CONCLUSIONS

The pilot results suggest that a health care-specific approach to conflict resolution can be effectively taught through facilitated practice, coaching, and feedback.

Simulated patients and objective structured clinical examinations: review of their use in medical education

Simulated or standardised patients have been used in medical education and other medical settings for some 30 years (Box 1). Their use encompasses undergraduate and postgraduate learning, the monitoring of doctors' performance and standardisation of clinical examinations. Simulation has been used for instruction in industry and the military for much longer (Jason et al, 1971) but the first known effective use of simulated patients was by Barrows & Abrahamson (1964), who used them to appraise students' performance in clinical neurology examinations.

Novel Transfer of Care Sign-out Assessment Tool in a Pediatric Emergency Department

OBJECTIVE

Transfer of care sign-outs (TOCS) for admissions from a pediatric emergency department have unique challenges. Standardized and reliable assessment tools for TOCS remain elusive. We describe the development, reliability, and validity of a TOCS assessment tool.

METHODS

Video recordings of resident TOCS were assessed to capture 4 domains: completeness, synopsis, foresight, and professionalism. In phase 1, 56 TOCS were used to modify the tool and improve reliability. In phase 2, 91 TOCS were used to examine validity. Analyses included Cronbach's alpha for internal structure, intraclass correlation and Cohen's kappa for interrater reliability, Pearson's correlation for relationships between variables, and 95% confidence interval of the mean for resident group comparisons.

RESULTS

Cronbach's alpha was 0.52 for internal structure of the tool's subjective rating scale. Intraclass correlation for the subjective rating scale items ranged from 0.70 to 0.80. Cohen's kappa for most objective checklist items ranged from 0.43 to 1. Content completeness was significantly correlated with synopsis, foresight, and professionalism (Pearson's r ranged from 0.36 to 0.62, P values were <0.001). House staff senior residents scored higher (on average) than interns and rotating senior residents in synopsis and foresight. Also, house staff interns scored higher (on average) than rotating senior residents in professionalism. House staff senior residents scored higher (on average) than rotating senior residents in content completeness.

CONCLUSIONS

We provide validity evidence to support using scores from the TOCS tool to assess higher-level transfer of care comprehension and communication by pediatric emergency department residents and to test interventions to improve TOCS.

Breaking bad news to patients with cancer: A randomized control trial of a brief communication skills training module incorporating the stories and preferences of actual patients

OBJECTIVE

This study tested the effectiveness of a brief, learner-centered, breaking bad news (BBN) communication skills training module using objective evaluation measures.

METHODS

This randomized control study (N=66) compared intervention and control groups of students (n=28) and residents' (n=38) objective structured clinical examination (OSCE) performance of communication skills using Common Ground Assessment and Breaking Bad News measures.

RESULTS

Follow-up performance scores of intervention group students improved significantly regarding BBN (colon cancer (CC), p=0.007, r=-0.47; breast cancer (BC), p=0.003, r=-0.53), attention to patient responses after BBN (CC, p<0.001, r=-0.74; BC, p=0.001, r=-0.65), and addressing feelings (BC, p=0.006, r=-0.48). At CC follow-up assessment, performance scores of intervention group residents improved significantly regarding BBN (p=0.004, r=-0.43), communication related to emotions (p=0.034, r=-0.30), determining patient's readiness to proceed after BBN and communication preferences (p=0.041, r=-0.28), active listening (p=0.011, r=-0.37), addressing feelings (p<0.001, r=-0.65), and global interview performance (p=0.001, r=-0.51).

CONCLUSION

This brief BBN training module is an effective method of improving BBN communication skills among medical students and residents.

PRACTICE IMPLICATIONS

Implementation of this brief individualized training module within health education programs could lead to improved communication skills and patient care.

Attachment styles and clinical communication performance in trainee doctors

OBJECTIVE

To investigate the relationship between trainee doctors' attachment style and their performance in qualifying clinical and communication skills assessments.

METHODS

Participants were 190 undergraduate medical students whose performance was assessed by examiners across two areas (communication and clinical skills) during their qualifying Objective Structured Clinical Examination (OSCE). Simulated patients also rated communication skills. Participants' attachment style was rated across two dimensions, avoidance and anxiety, using the Relationship Questionnaire (RQ).

RESULTS

Lower levels of attachment avoidance and anxiety significantly predicted higher performance in both communication and clinical skills.

CONCLUSION

Trainee doctors' attachment styles are associated with patient communication and clinical performance. Further research is needed to investigate the impact of attachment on consultations between doctors and patients within clinical settings.

PRACTICE IMPLICATION

Attachment theory can inform our understanding why, for some student doctors, interacting with patients may be particularly challenging and require additional support by medical educators.

Thrive or overload? The effect of task complexity on novices' simulation-based learning

CONTEXT

Fidelity is widely viewed as an important element of simulation instructional design based on its purported relationship with transfer of learning. However, higher levels of fidelity may increase task complexity to a point at which novices' cognitive resources become overloaded.

OBJECTIVES

In this experiment, we investigate the effects of variations in task complexity on novices' cognitive load and learning during simulation-based procedural skills training.

METHODS

Thirty-eight medical students were randomly assigned to simulation training on a simple or complex lumbar puncture (LP) task. Participants completed four practice trials on this task (skill acquisition). After 10 days of rest, all participants completed one additional trial on their assigned task (retention) and one trial on a 'very complex' simulation designed to be similar to the complex task (transfer). We assessed LP performance and cognitive load on each trial using multiple measures.

RESULTS

In both groups, LP performance improved significantly during skill acquisition (p ≤ 0.047, f = 0.29-0.96) and was maintained at retention. The simple task group demonstrated superior performance compared with the complex task group throughout these phases (p ≤ 0.002, d = 1.13-2.31). Cognitive load declined significantly in the simple task group (p < 0.009, f = 0.48-0.76), but not in the complex task group during skill acquisition, and remained lower at retention (p ≤ 0.024, d = 0.78-1.39). Between retention and transfer, LP performance declined and cognitive load increased in the simple task group, whereas both remained stable in the complex task group. At transfer, no group differences were observed in LP performance and cognitive load, except that the simple task group made significantly fewer breaches of sterility (p = 0.023, d = 0.80).

CONCLUSIONS

Reduced task complexity was associated with superior LP performance and lower cognitive load during skill acquisition and retention, but mixed results on transfer to a more complex task. These results indicate that task complexity is an important factor that may mediate (via cognitive overload) the relationship between instructional design elements (e.g. fidelity) and simulation-based learning outcomes.

Has the inclusion of a longitudinally integrated communication skills program improved consultation skills in medical students? A pilot study

Background:

Evidence highlights a lack of communication skills in doctors leading to dysfunctional consultations. To address this deficit, a private medical college instituted curricular reforms with inclusion of a longitudinal communication skills program. A pilot study was undertaken to evaluate the effectiveness of this program by comparing the consultation skills of medical students of this college with a medical college without a communication skills program.

Methods:

A 4-station Objective Structured Clinical Examination (OSCE) was conducted in the third and final year. Mann–Whitney U-test was used to compare the difference in the distribution between OSCE stations total and construct scores.

Results:

At the end of the third year, 21 (31.34%), students of the study site (medical college 1 [college with integrated longitudinal communication skills program]) and 31 (46.26%) students from the comparison site (medical college 2 [comparable college without communication skills program]) consented. Medical college 1 achieved a significantly higher overall mean total station score of 68.0% (standard deviation [SD] =13.5) versus 57.2% (SD = 15.4) (P < 0.001). Significantly higher mean scores were achieved on three stations. At the end of the final year, 19 students (29.3%) from medical college 1 and 22 (34%) students from medical college 2 consented. The difference in overall mean total station score reduced from 9.2% to 7.1% (70.2) (SD = 13.7) versus 63.1 (SD = 15.2) (P = 0.004). The mean scores of both colleges decreased in “Patient presenting with Hepatitis C Report” station (P values 0.004 and 0.775) and in “Patient Request for Faith Healing Therapy in Diabetes Mellitus” station (P values 0.0046 and 0.036), respectively.

Conclusion:

Longitudinal communication skills in an undergraduate curriculum positively impacted consultation skills. Community-based training and faculty development are required to develop effective patient-centered consultation skills.

Performance Assessment in Medical Education

The assessment of clinical competence is becoming increasingly complex, patient cen tered, and student driven. Traditionally, clini cal evaluation methods consisted primarily of faculty observations, oral examinations, and multiple-choice tests. Increased faculty work load, discontent with traditional methods of clinical skill assessment, and developments in the fields of psychology and education have led to the formation of new modalities, namely performance assessments. The literature per taining to the performance assessment with standardized patients is reviewed. Based on this literature, several areas for the future direction of performance assessment are pro posed, including (a) toward evidence-based locally developed assessments, (b) toward an understanding of educational outcomes and noncognitive assessment factors, and (c) toward more student-driven assessments.

Impact of focused training on communication skills of final-year medical students in a medical school in India

BACKGROUND

Although communication skills are important for a good physician-patient relationship, Indian medical curricula give very little emphasis on training medical students in this aspect.

AIMS

To determine the change in communication skills of final-year medical students following focused training.

METHODS

This was an educational interventional study done at Pondicherry Institute of Medical Sciences, a tertiary care teaching hospital in South India, to assess communication skills among final-year MBBS students. Fifty-two students (24 males and 28 females) participated in the study. A pre-test was conducted in the form of an objectively structured clinical examination (OSCE), followed by focused training for four hours. The same OSCE was administered as post-test. A comparison between the pre-test and post-test scores was done using Wilcoxon Signed Ranks Test.

RESULTS

Ninety-six per cent of participants (50 out of 52) showed improvement in their performance after the focused training. The mean marks of the pre-test and post-test were 10.77± 3 and 18.04±2, respectively, out of a maximum mark of 20 (p<0.05). One out of 52 participants did not show any improvement, and one participant scored less in the post-test compared to the pre-test. There was no significant difference in the performance between male and female students.

CONCLUSION

Focused training can enhance the communication skills of medical students. Hence, it may be included in the curriculum of undergraduate medical teaching programmes in India.

Reliability and validity of OSCE checklists used to assess the communication skills of undergraduate medical students: A systematic review

OBJECTIVES

To explore inter-rater agreement between reviewers comparing reliability and validity of checklist forms that claim to assess the communication skills of undergraduate medical students in Objective Structured Clinical Examinations (OSCEs).

METHODS

Papers explaining rubrics of OSCE checklist forms were identified from Pubmed, Embase, PsycINFO, and the ProQuest Education Databases up to 2013. Included were those studies that report empirical validity or reliability values for the communication skills assessment checklists used. Excluded were those papers that did not report reliability or validity.

RESULTS

Papers focusing on generic communication skills, history taking, physician-patient communication, interviewing, negotiating treatment, information giving, empathy and 18 other domains (ICC -0.12-1) were identified. Regarding the validity and reliability of the communication skills checklists, agreement between reviewers was 0.45.

CONCLUSIONS

Heterogeneity in the rubrics used in the assessment of communication skills and a lack of agreement between reviewers makes comparison of student competences within and across institutions difficult.

PRACTICE IMPLICATIONS

Consideration should be afforded to the adoption of a standardized measurement instrument to assess communication skills in undergraduate medical education. Future research will focus upon evaluating the potential impact of adoption of a standardized measurement instrument.

Examiner and simulated patient ratings of empathy in medical student final year clinical examination: are they useful?

BACKGROUND

Many medical schools state that empathy is important and have curricular learning outcomes covering its teaching. It is thought to be useful in team-working, good bedside manner, patient perspective taking, and improved patient care. Given this, one might expect it to be measured in assessment processes. Despite this, there is relatively little literature exploring how measures of empathy in final clinical examinations in medical school map onto other examination scores. Little is known about simulated patient (actors) rating of empathy in examinations in terms of inter-rater reliability compared with clinical assessors or correlation with overall examination results.

METHODS

Examiners in final year clinical assessments in one UK medical school rated 133 students on five constructs in Objective Structured Long Examination Record (OSLER) with real patients: gathering information, physical examination, problem solving, managing the diagnosis, and relationship with the patient. Scores were based on a standardized well-established penalty point system. In separate Objective Structured Clinical Examination (OSCE) stations, different examiners used the same penalty point system to score performance in both interactional and procedural stations. In the four interaction-based OSCE stations, examiners and simulated patient actors also independently rated empathy of the students.

RESULTS

The OSLER score, based on penalty points, had a correlation of -0.38 with independent ratings of empathy from the interactional OSCE stations. The intra-class correlation (a measure of inter-rater reliability) between the observing clinical tutor and ratings from simulated patients was 0.645 with very similar means. There was a significant difference between the empathy scores of the 94 students passing the first part of the sequential examination, based on combined OSCE and OSLER scores (which did not include the empathy scores), and 39 students with sufficient penalty points to trigger attendance for the second part (Cohen's d = 0.81).

CONCLUSIONS

These findings suggest that empathy ratings are related to clinical performance as measured by independent examiners. Simulated patient actors are able to give clinically meaningful assessment scores. This gives preliminary evidence that such empathy ratings could be useful for formative learning, and bolsters the call for more research to test whether they are robust enough to be used summatively.

How to assess communication, professionalism, collaboration and the other intrinsic CanMEDS roles in orthopedic residents: use of an objective structured clinical examination (OSCE)

BACKGROUND

Assessing residents' understanding and application of the 6 intrinsic CanMEDS roles (communicator, professional, manager, collaborator, health advocate, scholar) is challenging for postgraduate medical educators. We hypothesized that an objective structured clinical examination (OSCE) designed to assess multiple intrinsic CanMEDS roles would be sufficiently reliable and valid.

METHODS

The OSCE comprised 6 10-minute stations, each testing 2 intrinsic roles using case-based scenarios (with or without the use of standardized patients). Residents were evaluated using 5-point scales and an overall performance rating at each station. Concurrent validity was sought by correlation with in-training evaluation reports (ITERs) from the last 12 months and an ordinal ranking created by program directors (PDs).

RESULTS

Twenty-five residents from postgraduate years (PGY) 0, 3 and 5 participated. The interstation reliability for total test scores (percent) was 0.87, while reliability for each of the communicator, collaborator, manager and professional roles was greater than 0.8. Total test scores, individual station scores and individual CanMEDS role scores all showed a significant effect by PGY level. Analysis of the PD rankings of intrinsic roles demonstrated a high correlation with the OSCE role scores. A correlation was seen between ITER and OSCE for the communicator role, while the ITER medical expert and total scores highly correlated with the communicator, manager and professional OSCE scores.

CONCLUSION

An OSCE designed to assess the intrinsic CanMEDS roles was sufficiently valid and reliable for regular use in an orthopedic residency program.

Assessing medical students' communication skills by the use of standardized patients: emphasizing standardized patients' quality assurance

OBJECTIVE

The objective structured examination is one of the most valid, reliable, and effective tools for assessing clinical and communication skills, often by use of standard patients (SPs). SPs can also be assessors of those skills. One of the crucial areas when utilizing SP-based assessment is the quality and consistency assurance of their portrayal of the case and their ability to fill in checklists in an adequate way. The aim of this study was to assess the validity and reliability of SPs' ability to assess students' communication skill via a Calgary-Cambridge checklist.

METHOD

This cross-sectional and correlational study was conducted at the Tehran University of Medical Science. We first analyzed validity; the criterion validity of the SPs' filling in the checklists was assessed through determining the correlation between the SPs' completed checklists and the checklists filled in by three physician raters individually and then reproducibility: it was assessed by a test-retest approach inter-rater reliability.

RESULT

The mean correlation for assessing the validity of SPs' completed checklists by individual SPs was 0.81. The inter-rater reliability was calculated by kappa coefficient, and the total correlation among the three raters was 0.85. The reliability of the test-retest approach showed no significant differences between the test and re-test results.

CONCLUSION

The increased number of medical students and different faculties' responsibilities such as doing educational, research, and health services duties assessing medical student communication skills is a complex issue. The results of our study showed that trained SPs can be used as a valid tool to assess medical students' communication skills, which is also more cost effective and reduces work load of medical faculties.

Using a teaching OSCE to prompt learners to engage with patients who talk about religion and/or spirituality

PURPOSE

The objective structured clinical examination (OSCE) has only occasionally been used as a teaching tool. The authors describe the initial use of an educational innovation consisting of a teaching OSCE used as "sensitizing practice," followed by personal, guided, and group reflection.

METHOD

Staff and resident physicians and one medical student (N = 28) at a community hospital's family medicine residency participated in the innovation during August 2012. The initial use of the educational innovation allowed learners to engage in a potentially challenging conversation with a standardized patient about religion and/or spirituality (R/S). The aim of the innovation was not to equip learners with a particular tactic to introduce or discuss R/S but, rather, to prompt learners to engage in mindful practice with patients who identify R/S as part of their biopsychosocial contexts. Written, dyadic, and group reflection added value to the OSCE by allowing participants to reflect on a difficult learning objective over time.

RESULTS

Participants moved along the stages-of-change continuum when engaging in guided reflection compared with personal reflection. Additionally, all participants provided evidence of at least the preparation stage at the time of guided reflection. By following the OSCE's sensitizing practice with three periods of reflection, learners were enabled first, to recognize the need for readiness to address challenging communication topics (in this case, R/S) and, second, to reflect on practiced strategies for those conversations.

CONCLUSIONS

The educational innovation can help learners become more aware of and skillful in dealing with difficult physician-patient communication topics.

Designing and Implementing the Objective Structured Clinical Examination in Anesthesiology

Since its description in 1974, the Objective Structured Clinical Examination (OSCE) has gained popularity as an objective assessment tool of medical students, residents, and trainees. With the development of the anesthesiology residents’ milestones and the preparation for the Next Accreditation System, there is an increased interest in OSCE as an evaluation tool of the six core competencies and the corresponding milestones proposed by the Accreditation Council for Graduate Medical Education.

In this article the authors review the history of OSCE and its current application in medical education and in different medical and surgical specialties. They also review the use of OSCE by anesthesiology programs and certification boards in the United States and internationally. In addition, they discuss the psychometrics of test design and implementation with emphasis on reliability and validity measures as they relate to OSCE.

How to attain expertise in clinical communication?

Several factors complicate the attainment of expertise in clinical communication. Medical curricula and postgraduate training insufficiently provide the required learning conditions of deliberate practice to overcome these obstacles. In this paper we provide recommendations for learning objectives and teaching methods for the attainment of professional expertise in patient education. Firstly, we propose to use functional learning objectives derived from the goals and strategies of clinical communication. Secondly, we recommend using teaching and assessment methods which: (1) contain stimulating learning tasks with opportunities for immediate feedback, reflection and corrections, and (2) give ample opportunity for repetition, gradual refinements and practice in challenging situations. Video-on-the-job fits these requirements and can be used to improve the competency in patient education of residents and medical staff in clinical practice. However, video-on-the-job can only be successful if the working environment supports the teaching and learning of communication and if medical staff which supervises the residents, is motivated to improve their own communication and didactic skills.

Experience of clinical skills assessment in the Busan-Gyeongnam Consortium

PURPOSE

The purpose of this study is to judge the quality of clinical skills assessment in Busan-Gyeongnam Consortium.

METHODS

Fourth grade medical school students (n=350 in 2012 and n=419 in 2013) in the Busan-Gyeongnam Consortium were included in the study. The examination was consisted of 6 clinical performance examination (CPX) and 6 objective structured clinical examination (OSCE) stations. The students were divided into groups to take the exam in 4 sites during 3 days. The overall reliability was estimated by Cronbach alpha coefficient across the stations and the case reliability was by alpha across checklist items. Analysis of variance and between-group variation were used to evaluate the variation of examinee performance across different days and sites.

RESULTS

The mean total CPX/OSCE score was 67.0 points. The overall alpha across-stations was 0.66 in 2012 and 0.61 in 2013. The alpha across-items within a station was 0.54 to 0.86 in CPX, 0.51 to 0.92 in OSCE. There was no significant increase in scores between the different days. The mean scores over sites were different in 30 out of 48 stations but between-group variances were under 30%, except 2 cases.

CONCLUSION

The overall reliability was below 0.70 and standardization of exam sites was unclear. To improve the quality of exam, case development, item design, training of standardized patients and assessors, and standardization of sites are necessary. Above of all, we need to develop the well-organized matrix to measure the quality of the exam.

Inconsistency of residents' communication performance in challenging consultations

OBJECTIVE

Communication performance inconsistency between consultations is usually regarded as a measurement error that jeopardizes the reliability of assessments. However, inconsistency is an important phenomenon, since it indicates that physicians' communication may be below standard in some consultations.

METHODS

Fifty residents performed two challenging consultations. Residents' communication competency was assessed with the CELI instrument. Residents' background in communication skills training (CST) was also established. We used multilevel analysis to explore communication performance inconsistency between the two consultations. We also established the relationships between inconsistency and average performance quality, the type of consultation, and CST background.

RESULTS

Inconsistency accounted for 45.5% of variance in residents' communication performance. Inconsistency was dependent on the type of consultation. The effect of CST background training on performance quality was case specific. Inconsistency and average performance quality were related for those consultation combinations dissimilar in goals, structure, and required skills. CST background had no effect on inconsistency.

CONCLUSION

Physician communication performance should be of high quality, but also consistent regardless of the type and complexity of the consultation.

PRACTICE IMPLICATIONS

In order to improve performance quality and reduce performance inconsistency, communication education should offer ample opportunities to practice a wide variety of challenging consultations.

Assessing professionalism and ethics knowledge and skills: preferences of psychiatry residents

BACKGROUND

Professionalism is one of the fundamental expectations and a core competency in residency education. Although programs use a variety of evaluative methods, little is known about residents' views of and preferences regarding various methods of assessment.

METHOD

The authors surveyed residents at seven psychiatry residency programs in the United States regarding their attitudes on professionalism, ethics preparation, and evaluation in psychiatry residency training. This report describes their views on methods of assessing professionalism.

RESULTS

Residents strongly agreed that clinical supervision is an appropriate assessment method. Moreover, they rated clinical supervision more highly than oral examinations, short-answer questions, essays, and standardized patient interactions. Residents also strongly favored direct faculty observation of residents' interactions with actual patients and clinical team members.

CONCLUSION

This study suggests that both direct faculty supervision and other clinically-based assessments are methods accepted by psychiatry residents. Future research on the validity and effectiveness of these modes of assessment is needed.

Education in patient-physician communication: how to improve effectiveness?

OBJECTIVE

Despite educational efforts expertise in communication as required by the CanMEDS competency framework is not achieved by medical students and residents. Several factors complicate the learning of professional communication.

METHODS

We adapted the reflective-impulsive model of social behaviour to explain the complexities of learning professional communication behaviour. We formulated recommendations for the learning objectives and teaching methods of communication education. Our recommendations are based on the reflective-impulsive model and on the model of deliberate practice which complements the reflective-impulsive model. Our recommendations are substantiated by those we found in the literature.

RESULTS

The reflective-impulsive model explains why the results of communication education fall below expectations and how expertise in communication can be attained by deliberate practice. The model of deliberate practice specifies learning conditions which are insufficiently fulfilled in current communication programmes.

CONCLUSION

The implementation of our recommendations would require a great deal of effort. Therefore we doubt whether expertise in professional communication can be fully attained during medical training.

PRACTICE IMPLICATIONS

We propose that the CanMEDS communication competencies not be regarded as endpoints in medical education but as guidelines to improve communication competency through deliberate practice throughout a professional career.

Measurement precision for repeat examinees on a standardized patient examination

Examinees who initially fail and later repeat an SP-based clinical skills exam typically exhibit large score gains on their second attempt, suggesting the possibility that examinees were not well measured on one of those attempts. This study evaluates score precision for examinees who repeated an SP-based clinical skills test administered as part of the US Medical Licensing Examination sequence. Generalizability theory was used as the basis for computing conditional standard errors of measurement (SEM) for individual examinees. Conditional SEMs were computed for approximately 60,000 single-take examinees and 5,000 repeat examinees who completed the Step 2 Clinical Skills Examination(®) between 2007 and 2009. The study focused exclusively on ratings of communication and interpersonal skills. Conditional SEMs for single-take and repeat examinees were nearly indistinguishable across most of the score scale. US graduates and IMGs were measured with equal levels of precision at all score levels, as were examinees with differing levels of skill speaking English. There was no evidence that examinees with the largest score changes were measured poorly on either their first or second attempt. The large score increases for repeat examinees on this SP-based exam probably cannot be attributed to unexpectedly large errors of measurement.

Using an objective structured video exam to identify differential understanding of aspects of communication skills

BACKGROUND

Effective communication in health care is associated with patient satisfaction and improved clinical outcomes. Professional schools increasingly incorporate communication training into their curricula. The objective structured video exam (OSVE) is a video-based examination that provides an economical way of assessing students' knowledge of communication skills. This study presents a scoring strategy that enables blueprinting of an OSVE to consensus guidelines, to determine which aspects of communication skills create the most difficulty for students to understand and to what degree understanding improves through experiential communication skills training.

METHODS

Five interactions between a healthcare professional and client were scripted and filmed using standardized patients. The dialogues were mapped onto the Kalamazoo consensus statement by having five communication experts view each video and identify effective and ineffective use of communication skills. Undergraduate students enrolled in a communications course completed an OSVE on three occasions.

RESULTS

A total of 79 students completed at least one testing session. The scores assigned supported the validity of the scoring strategy as an indication of knowledge growth. Considerable variability was observed across Kalamazoo sub-domains.

CONCLUSION

With further refining, this scoring approach may prove useful for educators to tailor their education and assessment practices to specific consensus guidelines.

History and practice of competency-based assessment

CONTEXT

In response to historical trends in expectations of doctors, the goals of medical education are increasingly framed in terms of global competencies. The language of these competencies has tended to adopt a prescriptive, rather than descriptive, approach. However, despite widespread agreement on the importance of competency-based education and more than two decades of study, this effort has not generated a dependable set of assessment tools.

DISCUSSION

Because models of competency are legislated, rather than shaped by scholarly consideration of empirical data, it is unlikely that such models directly reflect actual human behaviour. Efforts to measure clinical behaviours could benefit from increased clarity in three related conceptual areas. Firstly, the language of educational constructs should be framed in terms of data-based hypotheses, rather than in terms of intuitively plausible abilities. Secondly, these constructs should be specified in terms of the situations to which they are relevant, rather than as global personal characteristics. Finally, the resources required to measure these constructs should be rigorously established because a common resource-based metric would allow for rational selection of assessment methods. Specific methods to establish each of these objectives are discussed.

CONCLUSIONS

The political process of negotiating educational objectives should not be confused with the scientific work of establishing coherent and interpretable patterns of behaviour. Although the two activities can complement one another, each has its own distinct methods and style of discourse. It is thus critical to maintain boundaries between these two approaches to defining professional performance.

Assessment of first-year veterinary students' communication skills using an objective structured clinical examination: the importance of context

Communication skills are considered to be a core clinical skill in veterinary medicine and essential for practice success, including outcomes of care for patients and clients. While veterinary schools include communication skills training in their programs, there is minimal knowledge on how best to assess communication competence throughout the undergraduate program. The purpose of this study was to further our understanding of the reliability, utility, and suitability of a communication skills Objective Structured Clinical Examination (OSCE). Specifically we wanted to (1) identify the greatest source of variability (student, rater, station, and track) within a first-year, four station OSCE using exam scores and scores from videotape review by two trained raters, and (2) determine the effect of different stations on students' communication skills performance. Reliability of the scores from both the exam data and the two expert raters was 0.50 and 0.46 respectively, with the greatest amount of variance attributable to student by station. The percentage of variance due to raters in the exam data was 16.35%, whereas the percentage of variance for the two expert raters was 0%. These results have three important implications. First, the results reinforce the need for communication educators to emphasize that use of communication skills is moderated by the context of the clinical interaction. Second, by increasing rater training the amount of error in the scores due to raters can be reduced and inter-rater reliability increases. Third, the communication assessment method (in this case the OSCE checklist) should be built purposefully, taking into consideration the context of the case.

A systematic review of the reliability of objective structured clinical examination scores

CONTEXT

The objective structured clinical examination (OSCE) is comprised of a series of simulations used to assess the skill of medical practitioners in the diagnosis and treatment of patients. It is often used in high-stakes examinations and therefore it is important to assess its reliability and validity.

METHODS

The published literature was searched (PsycINFO, PubMed) for OSCE reliability estimates (coefficient alpha and generalisability coefficients) computed either across stations or across items within stations. Coders independently recorded information about each study. A meta-analysis of the available literature was computed and sources of systematic variance in estimates were examined.

RESULTS

A total of 188 alpha values from 39 studies were coded. The overall (summary) alpha across stations was 0.66 (95% confidence interval [CI] 0.62-0.70); the overall alpha within stations across items was 0.78 (95% CI 0.73-0.82). Better than average reliability was associated with a greater number of stations and a higher number of examiners per station. Interpersonal skills were evaluated less reliably across stations and more reliably within stations compared with clinical skills.

CONCLUSIONS

Overall scores on the OSCE are often not very reliable. It is more difficult to reliably assess communication skills than clinical skills when considering both as general traits that should apply across situations. It is generally helpful to use two examiners and large numbers of stations, but some OSCEs appear more reliable than others for reasons that are not yet fully understood.

Standardisation of role players for the Clinical Skills Assessment of the MRCGP

This paper describes the processes used to prepare the role players who simulate the 'patients' in the MRCGP Clinical Skills Assessment (CSA) examination, one of the licensing qualifications for GPs completing their training. Because it is such a high-stakes exam, the CSA must be seen to be fair and trustworthy, and such attributes could be compromised by inadequate role player performance. On a CSA examination day, three role players simulate the same 'patient' simultaneously on three different exam floors, each playing the same case up to 26 times on the day, and these 'patients' must present in a consistent and reliable manner to ensure that all candidates face the same test situation. The paper describes the initial selection and training of the actors, how they are prepared on the day of the exam itself, and the quality assurance processes designed to ensure that the exam has excellent role players performing as the simulated 'patients'.

Medical and psychology students' self-assessed communication skills: A pilot study

OBJECTIVE

The purpose of this study was to describe how psychology and medical students assess their own competency and skills before and after training, in which role-play was used to teach interpersonal and communication skills.

METHOD

Interpersonal and communication skills were assessed with a semi-structured questionnaire before and after the training.

RESULTS

The students of both medicine and psychology estimated their skill levels to be higher after the course. The psychology students estimated their skills for communication, motivating interviewing, empathy and reflection, and change orientation to be better at the end of the course. Medical students estimated their communication skills, motivating interviewing skills, and change orientation skills to be better at the end of the course.

CONCLUSION

Even a short period of training in interpersonal and communication skills can positively affect the self-assessed skills of the medical students.

PRACTICE IMPLICATIONS

In the future, it would be worthwhile to pay attention to reflective teaching practices in the training of both medical and psychology students. The cognitive and emotional components of these practices help students to develop their own communication skills.