Discriminating Features of Narrative Evaluations of Communication Skills During an OSCE

R.I.M.E. and reason: multi-station OSCE enhancement to neutralize grade inflation

To offset grade inflation, many clerkships combine faculty evaluations with objective assessments including the Medical Examiners Subject Examination (NBME-SE) or Objective Structured Clinical Examination (OSCE), however, standardized methods are not established. Following a curriculum transition removing faculty clinical evaluations from summative grading, final clerkship designations of fail (F), pass (P), and pass-with-distinction (PD) were determined by combined NBME-SE and OSCE performance, with overall PD for the clerkship requiring meeting this threshold in both. At the time, 90% of students achieved PD on the Internal Medicine (IM) OSCE resulting in overall clerkship grades primarily determined by the NBME-SE. The clerkship sought to enhance the OSCE to provide a more thorough objective clinical skills assessment, offset grade inflation, and reduce the NBME-SE primary determination of the final clerkship grade. The single-station 43-point OSCE was enhanced to a three-station 75-point OSCE using the Reporter-Interpreter-Manager-Educator (RIME) framework to align patient encounters with targeted assessments of progressive skills and competencies related to the clerkship rotation. Student performances were evaluated pre- and post-OSCE enhancement. Student surveys provided feedback about the clinical realism of the OSCE and the difficulty. Pre-intervention OSCE scores were more tightly clustered (SD = 5.65%) around a high average performance with scores being highly negatively skewed. Post-intervention OSCE scores were more dispersed (SD = 6.88%) around a lower average with scores being far less skewed resulting in an approximately normal distribution. This lowered the total number of students achieving PD on the OSCE and PD in the clerkship, thus reducing the relative weight of the NMBE-SE in the overall clerkship grade. Student response was positive, indicating the examination was fair and reflective of their clinical experiences. Through structured development, OSCE assessment can provide a realistic and objective measurement of clinical performance as part of the summative evaluation of students.

Using Politeness Theory to Deconstruct how Pre-registrant Pharmacists approach Disagreement in the Workplace

OBJECTIVE

Using politeness theory, this study investigates how pre-registrant pharmacists engage in workplace disputes.

METHODS

Fifty six students participated in two mock job judgement scenarios. In Scenario A (n = 25), the conflict was with a colleague of lower hierarchical status, while in Scenario B (n = 32), the conflict was with a colleague of equal hierarchical status. Using politeness theory, responses were coded into three tiers: 1) engagement in the face threatening act (FTA); 2) use of on/off record approaches; and 3) communicative strategies. For tier 1, reasons for engaging in the FTA were also coded.

RESULTS

89% of the participants indicated they would unequivocally engage in the FTA. For Scenario A, the pharmacist's roles and responsibilities (40%) and for Scenario B, situational urgency/safety (65%) were the key drivers. Scenario A participants were more likely to use an on-record approach (80%) than those in Scenario B (68%). Empathetic approaches (56%) were more common in Scenario A, while explanatory approaches (64%) were more frequent in Scenario B.

CONCLUSION

This study shows that pre-registrant pharmacists are aware of the principles of face work. While the study only reflects what participants believe they would do, not their actual ability in professional disputes, it supports current research that students need simulated low-stakes opportunities to practise communication skills before entering the workplace, especially where situational urgency is present. Integration of politeness theory in workshop and feedback design could help students to link awareness to actual interaction, although the theory should be expanded to integrate urgency as a factor impacting on all levels of interactional decision-making.

An Applied Linguistics Study of How Students Prevent Embarrassments and Impositions During Interactive Examination OSCEs

OBJECTIVE

To assess the effectiveness of politeness strategies used by pharmacy students to avoid embarrassing or imposing on others during objective structured clinical examinations.

METHODS

A total of 19 objective structured clinical examinations video recordings of 10 students (participants) interacting with mock patients were examined using the framework of Politeness Theory (PT). All relevant participant acts (speech activities) were coded using PT into (1) type of face threatening acts (ie, potentially sensitive situations-as regarded by PT) and (2) politeness strategies used to mitigate them. Conversation Analysis was then used to examine the effectiveness of conversational strategies by judging the 'patient' responses to these strategies.

RESULTS

Most acts had the potential to impact patients' negative face needs (ie, desire to act autonomously, eg, upon the practitioner making a request), positive face needs (ie, desire to be liked, eg, upon the practitioner making a diagnosis), or both. Despite applying a variety of positive politeness strategies (eg, avoiding disagreement, or expressing understanding) to prevent embarrassment to the patient, and negative politeness strategies (eg, being indirect, using hedging, or minimizing the imposition) to avoid directly imposing on them, "dispreferred responses" showed participants mostly focused on avoiding impositions, corresponding to what they have been taught, rather embarrassments.

CONCLUSION

Participants were less aware that discussing sensitive topics could cause embarrassment to patients, with the potential to upset them. Developing teaching and evaluation methods to consider patients' face needs could help in assessing and improving pharmacy students' communication skills.

Factors influencing the rating of sonographer students' clinical performance

INTRODUCTION

Little is known about the factors influencing clinical supervisor-assessors' ratings of sonographer students' performance. This study identifies these influential factors and relates them to professional competency standards, with the aim of raising awareness and improving assessment practice.

METHODS

This study used archived written comments from 94 clinical assessors describing 174 sonographer students' performance one month into their initial clinical practice (2015-6). Qualitative mixed method analysis revealed factors influencing assessor ratings of student performance and provided an estimate of the valency, association, and frequency of these factors.

RESULTS

Assessors provided written comments for 93 % (n = 162/174) of students. Comments totaled 7190 words (mean of 44 words/student). One-third of comment paragraphs were wholly positive, two-thirds were equivocal. None were wholly negative. Thematic analysis revealed eleven factors, and eight sub-factors, influencing assessor impressions of five dimensions of performance. Of the factors mentioned, 84.6 % (n = 853/1008) related to professional competencies. While 15.4 % (n = 155/1008) were unrelated to competencies, instead reflecting humanistic factors such as student motivation, disposition, approach to learning, prospects and impact on supervisor and staff. Factors were prioritised and combined independently, although some associated.

CONCLUSION

Clinical assessors formed impressions based on student performance, humanistic behaviours and personal qualities not necessarily outlined in educational outcomes or professional competency standards. Their presence, and interrelations, impact success in clinical practice, through their contribution to, and indication of, competence.

IMPLICATIONS FOR PRACTICE

Sonographer student curricula and assessor training should raise awareness of the factors influencing performance ratings and judgement of clinical competence, particularly the importance of humanistic factors. Inclusion of narrative comments, multiple assessors, and broad performance dimensions would enhance clinical assessment of sonographer student performance.

Assessors' interpretations of narrative data on communication skills in a summative OSCE

OBJECTIVES

Increasingly, narrative assessment data are used to substantiate and enhance the robustness of assessor judgements. However, the interpretation of written assessment comments is inherently complex and relies on human (expert) judgements. The purpose of this study was to explore how expert assessors process and construe or bring meaning to narrative data when interpreting narrative assessment comments written by others in the setting of standardised performance assessment.

METHODS

Narrative assessment comments on student communication skills and communication scores across six objective structured clinical examination stations were obtained for 24 final-year pharmacy students. Aggregated narrative data across all stations were sampled for nine students (three good, three average and three poor performers, based on communication scores). A total of 10 expert assessors reviewed the aggregated set of narrative comments for each student. Cognitive (information) processing was captured through think-aloud procedures and verbal protocol analysis.

RESULTS

Expert assessors primarily made use of two strategies to interpret the narratives, namely comparing and contrasting, and forming mental images of student performance. Assessors appeared to use three different perspectives when interpreting narrative comments, including those of: (i) the student (placing him- or herself in the shoes of the student); (ii) the examiner (adopting the role of examiner and reinterpreting comments according to his or her own standards or beliefs), and (iii) the professional (acting as the profession's gatekeeper by considering the assessment to be a representation of real-life practice).

CONCLUSIONS

The present findings add to current understandings of assessors' interpretations of narrative performance data by identifying the strategies and different perspectives used by expert assessors to frame and bring meaning to written comments. Assessors' perspectives affect assessors' interpretations of assessment comments and are likely to be influenced by their beliefs, interpretations of the assessment setting and personal performance theories. These results call for the use of multiple assessors to account for variations in assessor perspectives in the interpretation of narrative assessment data.