Describing student performance: a comparison among clinical preceptors across cultural contexts

Understanding the Factors that Affect the Assessment of Student Performance in Pharmacy Practicums

OBJECTIVE

The purpose of this study was to explore factors that may influence a practice educator's assessment of an entry-to-practice pharmacy student during inpatient direct patient care practicums.

METHODS

This was a qualitative analysis of semi-structured interviews of existing practice educators from a variety of hospital practice environments. Participants were asked to assess a fictional case of a student's work, which provided a framework for a broader discussion of assessment practices. Interviews were transcribed and subjectively analyzed for themes and factors that each study participant considered for the case and in their past precepting experiences.

RESULTS

A total of 13 participants consented and were interviewed. Identified themes included the quality of student work, key aspects of the student's performance, professionalism, and the complexity of the patient assigned to the student. There was significant heterogeneity in both the assessment of the fictional student and the factors that influenced each participant's assessment. It was clear that not all guidance provided by the academic institution is read or followed. Participants described challenges in applying the assessment rubric, including lack of time, training, resources, knowledge of the degree requirements/structure, and psychological factors inherent in high-stakes courses.

CONCLUSION

Complex, varied, and often contradictory factors are used by experiential practice educators in their assessment of pharmacy students on practicum. These findings lead to inconsistency and heterogeneity when assessing a mock case and actual students. Educators should consider mandatory training to ensure those who assess students have an understanding of practicum course expectations and assessments.

Investigating preceptor experiences with cultural intelligence in pharmacy education

BACKGROUND

A culturally intelligent pharmacy workforce is critical for addressing health disparities and ensuring that healthcare teams are equipped to support the medication needs of patients. Despite the critical role of preceptors in developing aspiring pharmacists, little is known about how they create or manage cross-cultural situations for students.

OBJECTIVE

The objective of this study was to explore preceptor experiences teaching cultural intelligence within experiential pharmacy settings.

METHODS

A convergent parallel mixed methods approach was used with a 10-item survey measuring preceptor teaching self-efficacy (measured from 0-cannot do at all to 10-highly certain can do) and interviews/focus groups to further understand cultural intelligence teaching experiences. Data were analyzed according to the 4 domains of the cultural intelligence framework (i.e., cultural awareness, cultural knowledge, cultural practice, and cultural desire). Survey data were analyzed descriptively and qualitative data were analyzed deductively.

RESULTS

Participants (n = 24) were most confident Discussing factors underlying health and healthcare disparities (e.g., access, socioeconomic status, environment, racial/ethnic) (7.54 ± 2.04) and least confident in Understanding the importance of cultural desire in teaching students to be culturally intelligent healthcare practitioners (5.21 ± 2.72). All four cultural intelligence domains were identified in the qualitative data (n = 315 codes), with preceptors providing evidence of cultural awareness (n = 38, 12.1%), cultural knowledge (n = 54, 17.1%), cultural practice (n = 183 codes, 58.1%), and cultural desire (n = 40, 12.7%). Preceptors described various pedagogical strategies, such as case discussions, reflection, and simulation.

CONCLUSIONS

Participants provided insight into pedagogical strategies for cultural intelligence that could promote student learning in experiential settings and help explicate curricular gaps. Further research regarding applicability of the cultural intelligence framework is needed, including application of these strategies and opportunities for preceptor development.

Factors influencing the rating of sonographer students' clinical performance

INTRODUCTION

Little is known about the factors influencing clinical supervisor-assessors' ratings of sonographer students' performance. This study identifies these influential factors and relates them to professional competency standards, with the aim of raising awareness and improving assessment practice.

METHODS

This study used archived written comments from 94 clinical assessors describing 174 sonographer students' performance one month into their initial clinical practice (2015-6). Qualitative mixed method analysis revealed factors influencing assessor ratings of student performance and provided an estimate of the valency, association, and frequency of these factors.

RESULTS

Assessors provided written comments for 93 % (n = 162/174) of students. Comments totaled 7190 words (mean of 44 words/student). One-third of comment paragraphs were wholly positive, two-thirds were equivocal. None were wholly negative. Thematic analysis revealed eleven factors, and eight sub-factors, influencing assessor impressions of five dimensions of performance. Of the factors mentioned, 84.6 % (n = 853/1008) related to professional competencies. While 15.4 % (n = 155/1008) were unrelated to competencies, instead reflecting humanistic factors such as student motivation, disposition, approach to learning, prospects and impact on supervisor and staff. Factors were prioritised and combined independently, although some associated.

CONCLUSION

Clinical assessors formed impressions based on student performance, humanistic behaviours and personal qualities not necessarily outlined in educational outcomes or professional competency standards. Their presence, and interrelations, impact success in clinical practice, through their contribution to, and indication of, competence.

IMPLICATIONS FOR PRACTICE

Sonographer student curricula and assessor training should raise awareness of the factors influencing performance ratings and judgement of clinical competence, particularly the importance of humanistic factors. Inclusion of narrative comments, multiple assessors, and broad performance dimensions would enhance clinical assessment of sonographer student performance.

Workplace-Based Assessment in Cross-Border Health Professional Education

Construct: The globalization of healthcare has been accentuated by the export of health professional curricula overseas. Yet intact translation of pedagogies and practices devised in one cultural setting may not be possible or necessarily appropriate for alternate environments. Purposeful examination of workplace learning is necessary to understand how the source or "home" program may need adapting in the distributed or "host" setting. Background: Strategies to optimize cross-border medical education partnerships have been largely focused on elements of campus-based learning. Determining how host clinical supervisors approach assessment in experiential settings within a different culture and uphold the standards of home programs is relevant given the influence of context on trainees' demonstrated competencies. In this mixed-methods study, we sought to explore assessor judgments of student workplace-based performance made by preceptors sharing a pharmacy curriculum in Canada and Qatar. Approach: Using modified Delphi consensus technique, we asked clinical supervisors in Canada (n = 18) and in Qatar (n = 14) to categorize trainee performance as described in 16 student vignettes. The proportion of ratings for three levels of expectation (exceeds, meets, or below) was calculated and within-country group consensus achieved if the level of agreement reached 80%. Between-country group comparisons were measured using a chi-square statistic. We then conducted follow-up semi-structured interviews to gain further perspectives and clarify assessor rationale. Transcripts were analyzed using thematic content analysis. Results: The threshold for between-country group differences in assessor impressions was met for only two of the 16 student vignettes. Compared to Canadian clinical supervisors, relatively more preceptors in Qatar judged one described student as meets rather than exceeds expectations and one as meets rather than falls below expectations. Analysis of follow-up interviews exploring how culture may inform variations in assessor judgments identified themes associated with the profession, organization, learner, and supervisor performance theories but not their particular geographic context. Clinical supervisors in both countries were largely aligned in expectations of student knowledge, skills, and behaviors demonstrated in patient care and multidisciplinary team interactions. Conclusions: Our study demonstrated that variation in student assessment was more frequent among clinical supervisors within the same national context than any differences identified between the two countries. In these program settings, national sociocultural norms did not predict global assessor impressions or competency-specific judgments; instead, professional and organizational cultures were more likely to inform student characterizations of performance in workplace-based settings. Further study situated within the specific experiential learning contexts of cross-border health professional curricula is assuredly warranted.

Discriminating Features of Narrative Evaluations of Communication Skills During an OSCE

Construct: Authors examined the use of narrative comments for evaluation of student communications skills in a standardized, summative assessment (Objective Structured Clinical Examinations [OSCE]). Background: The use of narrative evaluations in workplace settings is gaining credibility as an assessment tool, but it is unknown how assessors convey judgments using narratives in high-stakes standardized assessments. The aim of this study was to explore constructs (i.e., performance dimensions), as well as linguistic strategies that assessors use to distinguish between poor and good students when writing narrative assessment comments of communication skills during an OSCE. Approach: Eighteen assessors from Qatar University were recruited to write narrative assessment comments of communication skills for 14 students completing a summative OSCE. Assessors scored overall communication performance on a 5-point scale. Narrative evaluations for the top and bottom 2 performing students for each station (based on communication scores) were analyzed for linguistic strategies and constructs that informed assessment decisions. Results: Seventy-two narrative evaluations with 662 comments were analyzed. Most comments (77%) were written without the use of politeness strategies. A further 22% of comments were hedged. Hedging was used more commonly in poor performers, compared to good performers (30% vs. 15%, respectively). Overarching constructs of confidence, adaptability, patient safety, and professionalism were key dimensions that characterized the narrative evaluations of students' performance. Conclusions: Results contribute to our understanding regarding the utility of narrative comments for summative assessment of communication skills. Assessors' comments could be characterized by the constructs of confidence, adaptability, patient safety, and professionalism when distinguishing between levels of student performance. Findings support the notion that judgments are arrived at by clustering sets of behaviors into overarching and meaningful constructs rather than by solely focusing on discrete behaviors. These results call for the development of better-anchored evaluation tools for communication assessment during OSCEs, constructively aligned with assessors' map of the reality of professional practice.

Pharmacy Preceptor Judgments of Student Performance and Behavior During Experiential Training

Objective. To report the findings of how Canadian preceptors perceive and subsequently evaluate diverse levels of trainees during pharmacy clerkships. Methods. Using modified Delphi technique, 17 Doctor of Pharmacy (PharmD) preceptors from across Canada categorized 16 student narrative descriptions pertaining to their perception of described student performance: exceeds, meets, or falls below their expectations. Results. Twelve (75%) student narratives profiles were categorized unanimously in the final round, six of which were below expectations. Out of 117 ratings of below expectations by responding preceptors, the majority (115, 98%) of post-baccalaureate PharmD students described would fail. Conversely, if the same narrative instead profiled a resident or an entry-to-practice PharmD student, rotation failure decreased to 95 (81%) and 89 (76%), respectively. Conclusion. Pharmacy preceptors do not uniformly judge the same described student performance and inconsistently apply failing rotation grades when they do agree that performance falls below expectations.