Finessing Feedback: Recommendations for Effective Feedback in the Emergency Department

Applying simulation learning theory to identify instructional strategies for Generation Z emergency medicine residency education

Introduction

Generation Z learners are entering emergency medicine (EM) residency training, bringing unique learning preferences that influence their engagement with residency education. To optimally teach and motivate this incoming generation of learners, EM educators must understand and adapt to the changing instructional landscape.

Methodology

The Simulation Leaders Advancing the Next Generation in Emergency Medicine (SLANG-EM) Workgroup was created to identify effective educational strategies for Generation Z learners entering EM. Members were faculty in the Society for Academic Emergency Medicine (SAEM) Simulation Academy, well versed in learning theory supporting simulation-based education (SBE) and actively involved in EM residency education.

Unique treatment/analysis

Through primary and secondary literature searches, the SLANG-EM Workgroup identified four distinctive learning preferences of Generation Z learners: (1) individualized and self-paced learning, (2) engaging and visual learning environments, (3) immediate and actionable feedback, and (4) combined personal and academic support. Workgroup members evaluated these learning preferences using a novel conceptual framework informed by the theoretical principles underpinning SBE, recommending instructional strategies for Generation Z EM residency learners across multiple educational environments.

Implications for educators

Instructional strategies were described for the didactic, simulation, and clinical learning environments. In the didactic environment, identified instructional strategies included meaningful asynchronous education, interactive small-group learning, and improved multimedia design. In the simulation environment, educational innovations particularly suitable for Generation Z learners included learner-centered debriefing, rapid-cycle deliberate practice, and virtual simulation. In the clinical environment, described instructional strategies involved setting learner-centered goals and delivering facilitative feedback in the context of an educational alliance. Overall, these instructional strategies were clustered around themes of student-centered education and the educator as facilitator, which align well with Generation Z learning preferences. These findings were synthesized and presented as an advanced workshop, "Delivering Effective Education to the Next Generation," at the 2023 SAEM Annual Meeting.

"Are They Listening, and Do They Find It Useful?"-Evaluation of Mid-Rotation Formative Subjective and Objective Feedback to Radiology Trainees

BACKGROUND

Residents commonly receive only end-of-rotation evaluations and thus are often unaware of their progress during a rotation. In 2021, our neuroradiology section instituted mid-rotation feedback in which rotating residents received formative subjective and objective feedback. The purpose of this study was to describe our feedback method and to evaluate if residents found it helpful.

METHODS

Radiology residents rotate 3-4 times on the neuroradiology service for 1-month blocks. At the midpoint of the rotation (2 weeks), 7-10 neuroradiology attendings discussed the rotating residents' subjective performance. One attending was tasked with facilitating this discussion and taking notes. Objective metrics were obtained from our dictation software. Compiled feedback was relayed to residents via email. A 16-question anonymous survey was sent to 39 radiology residents (R1-R4) to evaluate the perceived value of mid-rotation feedback. Odds ratios and 95% confidence intervals were computed using logistic regression.

RESULTS

Sixty-nine percent (27/39) of residents responded to the survey; 92.6% (25/27) of residents reported receiving mid-rotation feedback in ≥50% of neuroradiology rotations; 92.3% (24/26) of residents found the subjective feedback helpful; 88.4% (23/26) of residents reported modifying their performance as suggested (100% R1-R2 vs 70% R3-R4; OR: 15.4 CI:1.26, >30.0);59.1% (13/22) of residents found the objective metrics helpful (75% R1-R2 vs 40% R3-R4; OR: 3.92 CI:0.74, 24.39) and 68.2% (15/22) stated they modified their performance based on these metrics (83.3% R1-R2 vs 50.0% R3-R4; OR:4.2 CI:0.73, 30.55); and 84.6% (22/26) of residents stated that mid-rotation subjective feedback and 45.5% (10/22) stated that mid-rotation objective feedback should be implemented in other sections.

CONCLUSIONS

Majority of residents found mid-rotation feedback to be helpful in informing them about their progress and areas for improvement in the neuroradiology rotation, more so for subjective feedback than objective feedback. The majority of residents stated all rotations should provide mid-rotation subjective feedback.

The integrative feedback tool: assessing a novel feedback tool among emergency medicine residents

OBJECTIVE

Feedback is critical to the growth of learners. However, feedback quality can be variable in practice. Most feedback tools are generic, with few targeting emergency medicine. We created a feedback tool designed for emergency medicine residents, and this study aimed to evaluate the effectiveness of this tool.

METHODS

This was a single-center, prospective cohort study comparing feedback quality before and after introducing a novel feedback tool. Residents and faculty completed a survey after each shift assessing feedback quality, feedback time, and the number of feedback episodes. Feedback quality was assessed using a composite score from seven questions, which were each scored 1 to 5 points (minimum total score, 7 points; maximum, 35 points). Preintervention and postintervention data were analyzed using a mixed-effects model that took into account the correlation of random effects between study participants.

RESULTS

Residents completed 182 surveys and faculty members completed 158 surveys. The use of the tool was associated with improved consistency in the summative score of effective feedback attributes as assessed by residents (P=0.040) but not by faculty (P=0.259). However, most of the individual scores for attributes of good feedback did not reach statistical significance. With the tool, residents perceived that faculty spent more time providing feedback (P=0.040) and that the delivery of feedback was more ongoing throughout the shift (P=0.020). Faculty felt that the tool allowed for more ongoing feedback (P=0.002), with no perceived increase in the time spent delivering feedback (P=0.833).

CONCLUSION

The use of a dedicated tool may help educators provide more meaningful and frequent feedback without impacting the perceived required time needed to provide feedback.

Implementation of the SIMPL (Society for Improving Medical Professional Learning) performance assessment tool in the emergency department: A pilot study

Background

Feedback and assessment are difficult to provide in the emergency department (ED) setting despite their critical importance for competency-based education, and traditional end-of-shift evaluations (ESEs) alone may be inadequate. The SIMPL (Society for Improving Medical Professional Learning) mobile application has been successfully implemented and studied in the operative setting for surgical training programs as a point-of-care tool that incorporates three assessment scales in addition to dictated feedback. SIMPL may represent a viable tool for enhancing workplace-based feedback and assessment in emergency medicine (EM).

Methods

We implemented SIMPL at a 4-year EM residency program during a pilot study from March to June 2021 for observable activities such as medical resuscitations and related procedures. Faculty and residents underwent formal rater training prior to launch and were asked to complete surveys regarding the SIMPL app's content, usability, and future directions at the end of the pilot.

Results

A total of 36/58 (62%) of faculty completed at least one evaluation, for a total of 190 evaluations and an average of three evaluations per faculty. Faculty initiated 130/190 (68%) and residents initiated 60/190 (32%) evaluations. Ninety-one percent included dictated feedback. A total of 45/54 (83%) residents received at least one evaluation, with an average of 3.5 evaluations per resident. Residents generally agreed that SIMPL increased the quality of feedback received and that they valued dictated feedback. Residents generally did not value the numerical feedback provided from SIMPL. Relative to the residents, faculty overall responded more positively toward SIMPL. The pilot generated several suggestions to inform the optimization of the next version of SIMPL for EM training programs.

Conclusions

The SIMPL app, originally developed for use in surgical training programs, can be implemented for use in EM residency programs, has positive support from faculty, and may provide important adjunct information beyond current ESEs.

Developing Master Adaptive Learners: Implementation of a Coaching Program in Graduate Medical Education

n/a

Differences in faculty feedback for high, expected, and below-expected clinically performing emergency medicine residents

Objectives

A lifelong pursuit such as medicine is frequently paired with a framework of "deliberate practice" for improvement. It is unclear whether the quality of feedback varies across different learner levels. Our study aims to assess whether a difference exists in the quality of feedback delivered to high-, expected-, and below-expected performer emergency medicine (EM) residents based on their attending-identified performance level.

Methods

We conducted a retrospective review of written EM resident feedback collected between November 2018 and March 2021. Clinical performance level was subjectively determined by attending faculty in their feedback. Feedback was coded on a scale from 0-5 based on the presence (1) or absence (0) of the items modified from the Ende's SMART criteria: Specific (S), Measurable (M), Achievable (A), Relevant (R), and Time-bound (T). The primary outcome was any total modified SMART criteria score difference concerning performance level using logistic regression with Generalized Estimating Equations (GEE). Secondary outcomes were differences for individual criteria.

Results

We analyzed 1284 evaluations (311 high performers, 930 expected performers, and 43 below-expected performers) of 94 unique residents from 66 different evaluators. Mean total modified SMART scores were significantly higher in high and below-expected performers than those designated as expected performers by faculty evaluators. Achievable and Relevant written feedback was provided to high performers in a significantly larger proportion than expected and below-expected performers. Only 278 out of 1284 evaluations met criteria for Specific.

Conclusions

Mean total modified SMART feedback scores were significantly greater in high performers and below-expected performers when compared to expected performers. Achievable and Relevant feedback was provided in greater proportions to high performer residents compared to expected and below-expected performers. These findings are a challenge to academic faculty to engage in quality feedback delivery for residents at all performance levels.

Affecting Change: Enhancing Feedback Interactions with Radiology Trainees

Feedback is a critical part of the learning process and is a valuable tool to empower adult learners. Modern feedback theory places the learner at the center of the feedback encounter. Individual and institutional barriers to effective giving and receiving of feedback can be overcome through education and attention to the form and content of feedback. We review the elements of effective feedback and address issues of framing, environmental, and social factors which aid in providing psychological safety and trust, as necessary elements to create a culture of feedback in radiology training programs. We provide practical strategies to empower learners with the necessary skills to solicit, receive, and reflect on feedback.

The role of feedback in emergency ambulance services: a qualitative interview study

BACKGROUND

Several international studies suggest that the feedback that emergency ambulance service (EMS) personnel receive on the care they have delivered lacks structure, relevance, credibility and routine implementation. Feedback in this context can relate to performance or patient outcomes, can come from a variety of sources and can be sought or imposed. Evidence from health services research and implementation science, suggests that feedback can change professional behavior, improve clinical outcomes and positively influence staff mental health. The current study aimed to explore the experience of EMS professionals regarding current feedback provision and their views on how feedback impacts on patient care, patient safety and staff wellbeing.

METHODS

This qualitative study was conducted as part of a wider study of work-related wellbeing in EMS professionals. We used purposive sampling to select 24 frontline EMS professionals from one ambulance service in the United Kingdom and conducted semi-structured interviews. The data was analyzed in iterative cycles of inductive and deductive reasoning using Abductive Thematic Network Analysis. The analysis was informed by psychological theory, as well as models from the wider feedback effectiveness and feedback-seeking behavior literature.

RESULTS

Participants viewed current feedback provision as inadequate and consistently expressed a desire for increased feedback. Reported types of prehospital feedback included patient outcome feedback, patient-experience feedback, peer-to-peer feedback, performance feedback, feedforward: on-scene advice, debriefing and investigations and coroners' reports. Participants raised concerns that inadequate feedback could negatively impact on patient safety by preventing learning from mistakes. Enhancing feedback provision was thought to improve patient care and staff wellbeing by supporting personal and professional development.

CONCLUSIONS

In line with previous research in this area, this study highlights EMS professionals' strong desire for feedback. The study advances the literature by suggesting a typology of prehospital feedback and presenting a unique insight into the motives for feedback-seeking using psychological theory. A logic model for prehospital feedback interventions was developed to inform future research and development into prehospital feedback.

Confidence-competence alignment and the role of self-confidence in medical education: A conceptual review

CONTEXT

There have been significant advances in competency-based medical education (CBME) within health professions education. While most of the efforts have focused on competency, less attention has been paid to the role of confidence as a factor in preparing for practice. This paper seeks to address this deficit by exploring the role of confidence and the calibration of confidence with regard to competence.

METHODS

This paper presents a conceptual review of confidence and the calibration of confidence in different medical education contexts. Building from an initial literature review, the authors engaged in iterative discussions exploring divergent and convergent perspectives, which were then supplemented with targeted literature reviews. Finally, a stakeholder consultation was conducted to situate and validate the provisional findings.

RESULTS

A series of axioms were developed to guide perceptions and responses to different states of confidence in health professionals: (a) confidence can shape how we act and is optimised when it closely corresponds to reality; (b) self-confidence is task-specific, but also inextricably influenced by the individual self-conceptualisation, the surrounding system and society; (c) confidence is shaped by many external factors and the context of the situation; (d) confidence must be considered in conjunction with competence and (e) the confidence-competence ratio (CCR) changes over time. It is important to track learners' CCRs and work with them to maintain balance.

CONCLUSION

Confidence is expressed in different ways and is shaped by a variety of modifiers. While CBME primarily focuses on competency, proportional confidence is an integral component in ensuring safe and professional practice. As such, it is important to consider both confidence and competence, as well as their relationship in CBME. The CCR can serve as a key construct in developing mindful and capable health professionals. Future research should evaluate strategies for assessing CCR, identify best practices for teaching confidence and guiding self-calibration of CCR and explore the role of CCR in continuing professional development for individuals and teams.

Novel tool for assessing the quality of feedback in the emergency room (FEED-ER)

BACKGROUND

The Accreditation Council for Graduate Medical Education (ACGME) emphasizes constructive feedback as a critical component of residency training. Despite over a decade of using competency-based milestone evaluations, emergency medicine (EM) residency programs lack a standardized method for assessing the quality of feedback. We developed two novel EM-specific feedback surveys to assess the quality of feedback in the ER (FEED-ER) from both the resident and the faculty perspectives. This study aimed to evaluate the surveys' psychometric properties.

METHODS

We developed FEED-ER using a Likert scale with faculty and resident versions based on the ACGME framework and a literature review. The preliminary survey consisted of 25 questions involving the feedback domains of timeliness, respect/communication, specificity, action plan, and feedback culture. We conducted two modified Delphi rounds involving 17 content experts to ensure respondent understanding of the items, item coherence to corresponding feedback domains, thematic saturation of domain content, and time duration. A multicenter study was conducted at five university-based EDs in the United States and one in Thailand in 2019. We evaluated the descriptive statistics of the frequency of responses, validity evidence, and reliability of FEED-ER.

RESULTS

A total of 147 EM faculty and 126 EM residents completed the survey. Internal consistency was adequate (Cronbach's alpha > 0.70) and test-retest reliability showed adequate temporal stability (ICC > 0.80) for all dimensions. Content validity was deemed acceptable (CVC > 0.80) for all items. From the 25 items of FEED-ER, 23 loaded into the originally theorized dimensions (with factor loadings > 0.50). Additionally, the five feedback domains were found to be statistically distinct, with correlations between 0.40 and 0.60. The final survey has 23 items.

CONCLUSIONS

This is the first study to develop and provide validity evidence for an EM-specific feedback tool that has strong psychometric properties, is reproducible and reliable, and provides an objective measure for assessing the quality of feedback in the ED.

Direct Observation Tools in Emergency Medicine: A Systematic Review of the Literature

OBJECTIVES

Direct observation is important for assessing the competency of medical learners. Multiple tools have been described in other fields, although the degree of emergency medicine-specific literature is unclear. This review sought to summarize the current literature on direct observation tools in the emergency department (ED) setting.

METHODS

We searched PubMed, Scopus, CINAHL, the Cochrane Central Register of Clinical Trials, the Cochrane Database of Systematic Reviews, ERIC, PsycINFO, and Google Scholar from 2012 to 2020 for publications on direct observation tools in the ED setting. Data were dual extracted into a predefined worksheet, and quality analysis was performed using the Medical Education Research Study Quality Instrument.

RESULTS

We identified 38 publications, comprising 2,977 learners. Fifteen different tools were described. The most commonly assessed tools included the Milestones (nine studies), Observed Structured Clinical Exercises (seven studies), the McMaster Modular Assessment Program (six studies), Queen's Simulation Assessment Test (five studies), and the mini-Clinical Evaluation Exercise (four studies). Most of the studies were performed in a single institution, and there were limited validity or reliability assessments reported.

CONCLUSIONS

The number of publications on direct observation tools for the ED setting has markedly increased. However, there remains a need for stronger internal and external validity data.

Clinical Teaching: An Evidence-based Guide to Best Practices from the Council of Emergency Medicine Residency Directors

Clinical teaching is the primary educational tool use to train learners from day one of medical school all the way to the completion of fellowship. However, concerns over time constraints and patient census have led to a decline in bedside teaching. This paper provides a critical review of the literature on clinical teaching with a focus on instructor teaching strategies, clinical teaching models, and suggestions for incorporating technology. Recommendations for instructor-related teaching factors include adequate preparation, awareness of effective teacher attributes, using evidence-based-knowledge dissemination strategies, ensuring good communication, and consideration of environmental factors. Proposed recommendations for potential teaching strategies include the Socratic method, the One-Minute Preceptor model, SNAPPS, ED STAT, teaching scripts, and bedside presentation rounds. Additionally, this article will suggest approaches to incorporating technology into clinical teaching, including just-in-time training, simulation, and telemedical teaching. This paper provides readers with strategies and techniques for improving clinical teaching effectiveness.

Curated Collections for Educators: Five Key Papers about Receiving Feedback in Medical Education

Introduction Feedback is a complex, multi-component interaction that is essential for academic development and advancement. Successful feedback requires active involvement from both the giver and receiver. However, research and guidance on the subject mostly center on the role of the provider of feedback. But the receiver of feedback holds the true power in this interaction, choosing how to interpret the information and deciding whether or not to incorporate the feedback to instill behavioral change. In this article, the authors aim to summarize five key papers related to receiving feedback, in order to outline both relevant information for emerging clinician-educators and discern ways to use this information for faculty development. Methods In order to generate a list of key papers that describe the importance of receiving feedback, the authors conducted a consensus-building process informed by social media sources. Key articles on receiving feedback were aggregated through a literature search. This list was further augmented via an open call on Twitter for important papers regarding receiving feedback. Through these processes, a list of 43 papers was created on the topic of receiving feedback in medical education. After compiling this preliminary list, the authorship group engaged in a modified Delphi approach to build consensus on selecting papers that best described the process of receiving feedback. Results We present the group's five most highly rated papers on the topic of receiving feedback in medical education. These papers were deemed essential and have also been summarized based on their relevance to junior faculty members and faculty developers. Conclusion While giving and receiving feedback are both vital for growth and development, much of the research focuses solely on giving feedback. However, receiving feedback is equally, if not more, important for instilling change in the learner. We explore the power of receiving feedback in medical education through five key papers that analyze the subject. We believe these papers can serve as great learning resources for both junior faculty members and faculty developers. They can assist the junior faculty to cultivate the ability to receive feedback and also serve as resources to aid senior faculty in building faculty-development sessions.