Direct Observation Tools in Emergency Medicine: A Systematic Review of the Literature

The integrative feedback tool: assessing a novel feedback tool among emergency medicine residents

OBJECTIVE

Feedback is critical to the growth of learners. However, feedback quality can be variable in practice. Most feedback tools are generic, with few targeting emergency medicine. We created a feedback tool designed for emergency medicine residents, and this study aimed to evaluate the effectiveness of this tool.

METHODS

This was a single-center, prospective cohort study comparing feedback quality before and after introducing a novel feedback tool. Residents and faculty completed a survey after each shift assessing feedback quality, feedback time, and the number of feedback episodes. Feedback quality was assessed using a composite score from seven questions, which were each scored 1 to 5 points (minimum total score, 7 points; maximum, 35 points). Preintervention and postintervention data were analyzed using a mixed-effects model that took into account the correlation of random effects between study participants.

RESULTS

Residents completed 182 surveys and faculty members completed 158 surveys. The use of the tool was associated with improved consistency in the summative score of effective feedback attributes as assessed by residents (P=0.040) but not by faculty (P=0.259). However, most of the individual scores for attributes of good feedback did not reach statistical significance. With the tool, residents perceived that faculty spent more time providing feedback (P=0.040) and that the delivery of feedback was more ongoing throughout the shift (P=0.020). Faculty felt that the tool allowed for more ongoing feedback (P=0.002), with no perceived increase in the time spent delivering feedback (P=0.833).

CONCLUSION

The use of a dedicated tool may help educators provide more meaningful and frequent feedback without impacting the perceived required time needed to provide feedback.

Trends in emergency medicine resident procedural reporting over a 10-year period

Background

Procedural competency is expected of all emergency medicine (EM) residents upon graduation. The ACGME requires a minimum number of essential procedures to successfully complete training. However, data are limited on the actual number of procedures residents perform and prior studies are limited to single institutions over short time periods. This study sought to assess the number of Key Index Procedures completed during EM residency training and evaluate trends over time.

Methods

We conducted a retrospective review of graduating EM resident procedure logs across eight ACGME accredited residency programs over the last 10 years (2013-2022). Sites were selected to ensure diversity of program length, program type, and geography. All data from EM residents graduating in 2013-2022 were eligible for inclusion. Data from residents from combined training programs, those who did not complete their full training at that institution (i.e., transferred in/out), or those who did not have data available were excluded. We determined the list of procedures based upon the ACGME Key Index Procedures list. Sites obtained totals for each of the identified procedures for each resident upon graduation. We calculated the mean and 95% CI for each procedure.

Results

We collected data from a total of 914 residents, with 881 (96.4%) meeting inclusion criteria. The most common procedures were point-of-care ultrasound, adult medical resuscitation, adult trauma resuscitation, and intubation. The least frequent procedures included pericardiocentesis, cricothyroidotomy, cardiac pacing, vaginal delivery, and chest tubes. Most procedures were stable over time with the exception of lumbar punctures (decreased) and point-of-care ultrasound (increased).

Conclusions

In a national sample of EM programs, procedural numbers remained stable except for lumbar puncture and ultrasound. This information can inform residency training curricula and accreditation requirements.

Exploring the use of rating scales with entrustment anchors in workplace-based assessment

PURPOSE

Competency-based medical education (CBME) has prompted widespread implementation of workplace-based assessment (WBA) tools using entrustment anchors. This study aimed to identify factors that influence faculty's rating choices immediately following assessment and explore their experiences using WBAs with entrustment anchors, specifically the Ottawa Surgical Competency Operating Room Evaluation scale.

METHOD

A convenience sample of 50 semi-structured interviews with Emergency Medicine (EM) physicians from a single Canadian hospital were conducted between July and August 2019. All interviews occurred within two hours of faculty completing a WBA of a trainee. Faculty were asked what they considered when rating the trainee's performance and whether they considered an alternate rating. Two team members independently analysed interview transcripts using conventional content analysis with line-by-line coding to identify themes.

RESULTS

Interviews captured interactions between 70% (26/37) of full-time EM faculty and 86% (19/22) of EM trainees. Faculty most commonly identified the amount of guidance the trainee required as influencing their rating. Other variables such as clinical context, trainee experience, past experiences with the trainee, perceived competence and confidence were also identified. While most faculty did not struggle to assign ratings, some had difficulty interpreting the language of entrustment anchors, being unsure whether their assessment should be retrospective or prospective in nature, and if/how the assessment should change whether they were 'in the room' or not.

CONCLUSIONS

By going to the frontline during WBA encounters, this study captured authentic and honest reflections from physicians immediately engaged in assessment using entrustment anchors. While many of the factors identified are consistent with previous retrospective work, we highlight how some faculty consider factors outside the prescribed approach and struggle with the language of entrustment anchors. These results further our understanding of 'in-the-moment' assessments using entrustment anchors and may facilitate effective faculty development regarding WBA in CBME.

The standardized letter of evaluation in emergency medicine: Are the qualifications useful?

OBJECTIVES

The standardized letter of evaluation (SLOE) in emergency medicine (EM) is a widely used metric for determining interview invitations and ranking of candidates. Previous research has questioned the validity of certain sections of the SLOE. However, there remains a paucity of literature on the qualifications for EM section, which evaluates seven attributes of applicants. The aim of this study was to determine the correlation between the qualifications questions and grades, global assessment, and anticipated rank list position for EM applicants.

METHODS

A multi-institutional cross-sectional study was performed using SLOEs from applicants to three geographically distinct U.S. EM residency programs during the 2019-2020 application cycle. We abstracted EM rotation grade, qualifications scores, global assessment, and anticipated rank list position from the SLOEs. A Spearman correlation was calculated between each of the qualifications scores and the applicant's grades, global assessment, and anticipated rank list position in a pairwise fashion.

RESULTS

In total, 2,106 unique applicants (4,939 SLOEs) were included. Of the seven qualifications for EM questions, three were moderately to strongly correlated with global assessment and anticipated rank list position: "ability to develop and justify an appropriate differential and a cohesive treatment plan" (ρ = 0.65 and ρ = 0.63, respectively; p < 0.001), "how much guidance do you predict this applicant will need during residency?" (ρ = 0.68 and ρ = 0.68, respectively; p < 0.001), and "what is your prediction of success for the applicant?" (ρ = 0.69 and ρ = 0.69, respectively; p < 0.001). There was no strong correlation between the seven qualifications and grades.

CONCLUSIONS

There was a moderate to strong correlation between three of seven qualifications for EM questions (ability to develop and justify a differential and develop a cohesive plan, anticipated need for the amount of guidance, and prediction of success) with both global assessment and anticipated rank list position, suggesting that these qualifications may provide the most useful data to residency selection while some of the other factors may not be needed.

Workplace-based Assessment Data in Emergency Medicine: A Scoping Review of the Literature

OBJECTIVE

In the era of competency-based medical education (CBME), the collection of more and more trainee data is being mandated by accrediting bodies such as the Accreditation Council for Graduate Medical Education and the Royal College of Physicians and Surgeons of Canada. However, few efforts have been made to synthesize the literature around the current issues surrounding workplace-based assessment (WBA) data. This scoping review seeks to synthesize the landscape of literature on the topic of data collection and utilization for trainees' WBAs in emergency medicine (EM).

METHODS

The authors conducted a scoping review in the style of Arksey and O'Malley, seeking to synthesize and map literature on collecting, aggregating, and reporting WBA data. The authors extracted, mapped, and synthesized literature that describes, supports, and substantiates effective data collection and utilization in the context of the CBME movement within EM.

RESULTS

Our literature search retrieved 189 potentially relevant references (after removing duplicates) that were screened to 29 abstracts and papers relevant to collecting, aggregating, and reporting WBAs. Our analysis shows that there is an increasing temporal trend toward contributions in these topics, with the majority of the papers (16/29) being published in the past 3 years alone.

CONCLUSION

There is increasing interest in the areas around data collection and utilization in the age of CBME. The field, however, is only beginning to emerge, leaving more work that can and should be done in this area.