Workplace-based Assessment Data in Emergency Medicine: A Scoping Review of the Literature

Automated Analysis of Preceptor Comments: A Pilot Study Using Sentiment Analysis to Identify Potential Student Issues in Experiential Education

OBJECTIVE

The purpose of this paper is to describe a sentiment analysis program that aids in identifying pharmacy students at risk for progression issues by automatically scoring preceptor comments as positive or negative.

METHODS

An R-based program to analyze advanced pharmacy practice experiences and introductory pharmacy practice experiences midpoint evaluation of preceptor comments was piloted in phase 1 by comparing the sentiment analysis algorithm results to human coding. The algorithm was refined in phase 2. In phase 3, the validation phase, the final sentiment analysis algorithm analyzed all midpoint student evaluations (n = 1560). Sentiment scores were generated for each preceptor comment, and correlations were performed between sentiment scores and the quantitative scoring provided on the assessment.

RESULTS

In phase 1, agreement between faculty coders and sentiment analysis was 96%, and in phase 2, agreement between the final codes and sentiment analysis was 92.4% once keywords were added to the sentiment dictionary. In phase 3, a total of 3919 comments from 1560 evaluations were analyzed, and overall, the sentiment analysis results aligned with the quantitative data.

CONCLUSION

This sentiment analysis algorithm was accurate in capturing positive and negative comments corresponding to pharmacy student performance. Given the accuracy of this preliminary validation for flagging preceptor comments, there are numerous implications when considering the use of sentiment analysis in pharmacy education. Using a sentiment analysis program minimizes the number of qualitative preceptor comments needing review by experiential faculty, as this program can aid in identifying students at risk of progression issues.

Examining enablers and barriers to entrustable professional activity acquisition using the theoretical domains framework: A qualitative framework analysis study

Background

Without a clear understanding of the factors contributing to the effective acquisition of high-quality entrustable professional activity (EPA) assessments, trainees, supervising faculty, and training programs may lack appropriate strategies for successful EPA implementation and utilization. The purpose of this study was to identify barriers and facilitators to acquiring high-quality EPA assessments in Canadian emergency medicine (EM) training programs.

Methods

We conducted a qualitative framework analysis study utilizing the Theoretical Domains Framework (TDF). Semistructured interviews of EM resident and faculty participants underwent audio recording, deidentification, and line-by-line coding by two authors, being coded to extract themes and subthemes across the domains of the TDF.

Results

From 14 interviews (eight faculty and six residents) we identified, within the 14 TDF domains, major themes and subthemes for barriers and facilitators to EPA acquisition for both faculty and residents. The two most cited domains (and their frequencies) among residents and faculty were environmental context and resources (56) and behavioral regulation (48). Example strategies to improving EPA acquisition include orienting residents to the competency-based medical education (CBME) paradigm, recalibrating expectations relating to "low ratings" on EPAs, engaging in continuous faculty development to ensure familiarity and fluency with EPAs, and implementing longitudinal coaching programs between residents and faculty to encourage repetitive longitudinal interactions and high-quality specific feedback.

Conclusions

We identified key strategies to support residents, faculty, programs, and institutions in overcoming barriers and improving EPA assessment processes. This is an important step toward ensuring the successful implementation of CBME and the effective operationalization of EPAs within EM training programs.

Do entrustment scales make a difference in the inter-rater reliability of the workplace-based assessment?

BACKGROUND

A workplace-based assessment (WBA) is used to assess learners' competencies in their workplaces. Many workplace assessment tools are available and validated to assess various constructs. The implementation of workplace-based assessment requires proper training of the staff.

OBJECTIVE

This study aimed to explore the impact of staff training on WBA practices and evaluate the inter-rater reliability of these practices while using entrustment scales, performance descriptors, and personal judgment.

DESIGN

A quasi-experimental study, in which the staff members of the orthopedic department were invited to participate in a training program on the use of entrustment scales and assessment descriptors within the WBA tools. As a response to the training, subjective judgment was replaced by entrustment scales and performance descriptors in a trauma course offered by the orthopedic department. The inter-rater reliability of the WBA was evaluated using various rating scales.

RESULTS

The entrustment scales had higher inter-rater reliability of the assessment tools than performance descriptors and the personal judgment.

CONCLUSION

The inter-rater reliability was highest when using entrustment scales for WBAs, which could indicate that the entrustment scales achieve good psychometric properties as regards consistency among different raters. Thus, they decrease the confounding effect of differences in assessors. They may also give a clearer image of the actual academic level of the learners.

Implementation of competence committees during the transition to CBME in Canada: A national fidelity-focused evaluation

PURPOSE

This study evaluated the fidelity of competence committee (CC) implementation in Canadian postgraduate specialist training programs during the transition to competency-based medical education (CBME).

METHODS

A national survey of CC chairs was distributed to all CBME training programs in November 2019. Survey questions were derived from guiding documents published by the Royal College of Physicians and Surgeons of Canada reflecting intended processes and design.

RESULTS

Response rate was 39% (113/293) with representation from all eligible disciplines. Committee size ranged from 3 to 20 members, 42% of programs included external members, and 20% included a resident representative. Most programs (72%) reported that a primary review and synthesis of resident assessment data occurs prior to the meeting, with some data reviewed collectively during meetings. When determining entrustable professional activity (EPA) achievement, most programs followed the national specialty guidelines closely with some exceptions (53%). Documented concerns about professionalism, EPA narrative comments, and EPA entrustment scores were most highly weighted when determining resident progress decisions.

CONCLUSIONS

Heterogeneity in CC implementation likely reflects local adaptations, but may also explain some of the variable challenges faced by programs during the transition to CBME. Our results offer educational leaders important fidelity data that can help inform the larger evaluation and transformation of CBME.

The standardized letter of evaluation in emergency medicine: Are the qualifications useful?

OBJECTIVES

The standardized letter of evaluation (SLOE) in emergency medicine (EM) is a widely used metric for determining interview invitations and ranking of candidates. Previous research has questioned the validity of certain sections of the SLOE. However, there remains a paucity of literature on the qualifications for EM section, which evaluates seven attributes of applicants. The aim of this study was to determine the correlation between the qualifications questions and grades, global assessment, and anticipated rank list position for EM applicants.

METHODS

A multi-institutional cross-sectional study was performed using SLOEs from applicants to three geographically distinct U.S. EM residency programs during the 2019-2020 application cycle. We abstracted EM rotation grade, qualifications scores, global assessment, and anticipated rank list position from the SLOEs. A Spearman correlation was calculated between each of the qualifications scores and the applicant's grades, global assessment, and anticipated rank list position in a pairwise fashion.

RESULTS

In total, 2,106 unique applicants (4,939 SLOEs) were included. Of the seven qualifications for EM questions, three were moderately to strongly correlated with global assessment and anticipated rank list position: "ability to develop and justify an appropriate differential and a cohesive treatment plan" (ρ = 0.65 and ρ = 0.63, respectively; p < 0.001), "how much guidance do you predict this applicant will need during residency?" (ρ = 0.68 and ρ = 0.68, respectively; p < 0.001), and "what is your prediction of success for the applicant?" (ρ = 0.69 and ρ = 0.69, respectively; p < 0.001). There was no strong correlation between the seven qualifications and grades.

CONCLUSIONS

There was a moderate to strong correlation between three of seven qualifications for EM questions (ability to develop and justify a differential and develop a cohesive plan, anticipated need for the amount of guidance, and prediction of success) with both global assessment and anticipated rank list position, suggesting that these qualifications may provide the most useful data to residency selection while some of the other factors may not be needed.