Attending and resident satisfaction with feedback in the emergency department

Through the Looking Glass: Comparing Hospitalists' and Internal Medicine Residents' Perceptions of Feedback

INTRODUCTION

Feedback is critical for resident growth and is most effective when the relationship between residents and attendings is collaborative, with shared expectations for the purpose, timing, and manner of communication for feedback. Within internal medicine, there is limited work exploring the resident and hospitalist perspectives on whether key elements are included in feedback sessions.

METHODS

We surveyed internal medicine residents and supervising hospitalists at a large urban training program about their perspectives on four components of effective feedback: specificity,timeliness, respectful communication, and actionability.

RESULTS

We received surveys from 130/184 internal medicine residents and 74/129 hospitalists (71% and 57% response rate, respectively). Residents and hospitalists differed in their perspectives about specificity and timeliness: 54% (70/129) of residents reported they did not receive specific feedback while 90% (65/72) of hospitalists reported they delivered specific feedback (p<0.01), and 33% (43/129) of residents compared with 82% (59/72) of hospitalists perceived feedback as timely (p<0.01). Internal medicine residents and hospitalists reported concordant rates of feedback sessions consisting of a two-way conversation (84%, 109/129; 89%, 64/72, respectively, p=0.82) and that communication was delivered in a respectful manner (95%, 122/129; 97%, 70/72, respectively, p=0.57).

CONCLUSIONS

We observed discordance between internal medicine residents and supervising hospitalist perspectives on the inclusion of two critical components of feedback: specificity and timing. The hospitalist cohort reported delivering more components of effective feedback than the resident cohort reported receiving. The etiology of this discordance is likely multifactorial and requires further investigation.

The feedback dilemma in medical education: insights from medical residents' perspectives

BACKGROUND

Feedback is a critical component of the learning process in a clinical setting. This study aims to explore medical residents' perspectives on feedback delivery and identify potential barriers to feedback-seeking in clinical training.

METHODS

This cross-sectional study involved 180 medical residents across seventeen specialties. We employed the validated Residency Education Feedback Level Evaluation in Clinical Training (REFLECT) tool to assess residents' perspectives on their attitude toward feedback, quality of feedback, perceived importance, and reaction to feedback. Additionally, we explored barriers to feedback-seeking behavior among medical residents.

RESULTS

The majority of medical residents held positive attitudes toward feedback. They agreed that feedback improves their clinical performance (77.7%), professional behavior (67.2%), and academic motivation (56.7%), while also influencing them to become a better specialist in their future career (72.8%). However, the study revealed critical deficiencies in the feedback process. Only 25.6% of residents reported receiving regular feedback and less than half reported that feedback was consistently delivered at suitable times and locations, was sufficiently clear or included actionable plans for improvement. A minority (32.2%) agreed that faculty had sufficient skills to deliver feedback effectively. Moreover, peer-to-peer feedback appeared to be a primary source of feedback among residents. Negative feedback, though necessary, often triggered feelings of stress, embarrassment, or humiliation. Notably, there were no significant differences in feedback perceptions among different specialties. The absence of a feedback-seeking culture emerged as a central barrier to feedback-seeking behavior in the clinical setting.

CONCLUSIONS

Establishing shared expectations and promoting a culture of feedback-seeking could bridge the gap between residents' perceptions and faculty feedback delivery. Furthermore, recognizing the role of senior and peer residents as valuable feedback sources can contribute to more effective feedback processes in clinical training, ultimately benefiting resident development and patient care.

Feedback perceptions of first year medical residents: An intervention-based survey study

BACKGROUND

Feedback in residency is a necessity for progression toward clinical competency and is included in The Accreditation Council for Graduate Medical Education (ACGME) milestones as an essential component for accreditation.

PURPOSE

Our study elucidates perceptions of feedback of first-year residents and aims to identify how these perceptions change after education on building expertise through deliberate practice.

METHODS

First-year internal medicine and neurology residents of a mid-sized university-affiliated residency program answered a five-question 5-point unipolar response scale questionnaire regarding feedback perceptions before and after attending a workshop about building expertise through effective feedback during residency orientation. Related-Samples Wilcoxon Signed Rank Test was applied for comparing pre- versus post-questionnaire data.

RESULTS

Of 31 first-year residents, 29 completed the pre-questionnaire for a completion rate of 93.5%, while 24 of 31 completed the post-questionnaire for a completion rate of 77.4%. Of the five questions, three improved when comparing pre and post responses to, including the questions on confidence in the ability to procure feedback (p = <0.001), the effort put into procuring feedback (p = 0.001), and frequency of seeking feedback (p = 0.002). Interest in receiving feedback and the importance of feedback remained unchanged after workshop attendance.

CONCLUSION

Residents should be educated on building expertise through deliberate practice and how to obtain high-quality feedback, given the emphasis and essentiality of feedback within the milestone assessment system and the core competencies of ACGME. In our study, education on these topics led to significant improvement in resident perceptions of confidence in the ability to procure feedback, effort put into procuring feedback, and frequency at which feedback would be sought.

Preparing for the PICU: A Qualitative Study of Residents as They Prepare for Their First Pediatric Critical Care Rotation

Learning critical care medicine in the pediatric intensive care unit (PICU) can be stressful. Through semistructured interviews ( n  = 16), this study explored the emotions, perceptions, and motivations of pediatric medicine (PM) and emergency medicine (EM) residents, as they prepared for their first PICU rotation. Qualitative data were collected and analyzed using the grounded theory method. Three resultant themes emerged: (1) residents entered the PICU with a range of intense emotions and heightened expectations; (2) they experienced prior history of psychologically traumatic learning events (adverse learning experiences or ALEs); and (3) informed by ALEs, residents prepared for their rotation by focusing heavily on their most basic level of physiological needs and adopting a survival mindset prior to the start of the rotation. These three themes led to a substantive, or working, theory that ALE-associated events may affect how residents approach upcoming learning opportunities. Consequently, adapting a trauma-informed approach as a component of medical education may improve resident learning experiences in the PICU and beyond.

Developing residents' feedback literacy in emergency medicine: Lessons from design-based research

Objectives

Residents in emergency medicine have reported dissatisfaction with feedback. One strategy to improve feedback is to enhance learners' feedback literacy-i.e., capabilities as seekers, processors, and users of performance information. To do this, however, the context in which feedback occurs needs to be understood. We investigated how residents typically engage with feedback in an emergency department, along with the potential opportunities to improve feedback engagement in this context. We used this information to develop a program to improve learners' feedback literacy in context and traced the reported translation to practice.

Methods

We conducted a year-long design-based research study informed by agentic feedback principles. Over five cycles in 2019, we interviewed residents and iteratively developed a feedback literacy program. Sixty-six residents participated and data collected included qualitative evaluation surveys (n = 55), educator-written reflections (n = 5), and semistructured interviews with residents (n = 21). Qualitative data were analyzed using framework analysis.

Results

When adopting an agentic stance, residents reported changes to the frequency and tenor of their feedback conversations, rendering the interactions more helpful. Despite reporting overall shifts in their conceptions of feedback, they needed to adjust their feedback engagement depending on changing contextual factors such as workload. These microsocial adjustments suggest their feedback literacy develops through an interdependent process of individual intention for feedback engagement-informed by an agentic stance-and dynamic adjustment in response to the environment.

Conclusions

Resident feedback literacy is profoundly contextualized, so developing feedback literacy in emergency contexts is more nuanced than previously reported. While feedback literacy can be supported through targeted education, our findings raise questions for understanding how emergency medicine environments afford and constrain learner feedback engagement. Our findings also challenge the extent to which this contextual feedback know-how can be "developed" purposefully outside of the everyday work.

Implementation of the SIMPL (Society for Improving Medical Professional Learning) performance assessment tool in the emergency department: A pilot study

Background

Feedback and assessment are difficult to provide in the emergency department (ED) setting despite their critical importance for competency-based education, and traditional end-of-shift evaluations (ESEs) alone may be inadequate. The SIMPL (Society for Improving Medical Professional Learning) mobile application has been successfully implemented and studied in the operative setting for surgical training programs as a point-of-care tool that incorporates three assessment scales in addition to dictated feedback. SIMPL may represent a viable tool for enhancing workplace-based feedback and assessment in emergency medicine (EM).

Methods

We implemented SIMPL at a 4-year EM residency program during a pilot study from March to June 2021 for observable activities such as medical resuscitations and related procedures. Faculty and residents underwent formal rater training prior to launch and were asked to complete surveys regarding the SIMPL app's content, usability, and future directions at the end of the pilot.

Results

A total of 36/58 (62%) of faculty completed at least one evaluation, for a total of 190 evaluations and an average of three evaluations per faculty. Faculty initiated 130/190 (68%) and residents initiated 60/190 (32%) evaluations. Ninety-one percent included dictated feedback. A total of 45/54 (83%) residents received at least one evaluation, with an average of 3.5 evaluations per resident. Residents generally agreed that SIMPL increased the quality of feedback received and that they valued dictated feedback. Residents generally did not value the numerical feedback provided from SIMPL. Relative to the residents, faculty overall responded more positively toward SIMPL. The pilot generated several suggestions to inform the optimization of the next version of SIMPL for EM training programs.

Conclusions

The SIMPL app, originally developed for use in surgical training programs, can be implemented for use in EM residency programs, has positive support from faculty, and may provide important adjunct information beyond current ESEs.

Differences in faculty feedback for high, expected, and below-expected clinically performing emergency medicine residents

Objectives

A lifelong pursuit such as medicine is frequently paired with a framework of "deliberate practice" for improvement. It is unclear whether the quality of feedback varies across different learner levels. Our study aims to assess whether a difference exists in the quality of feedback delivered to high-, expected-, and below-expected performer emergency medicine (EM) residents based on their attending-identified performance level.

Methods

We conducted a retrospective review of written EM resident feedback collected between November 2018 and March 2021. Clinical performance level was subjectively determined by attending faculty in their feedback. Feedback was coded on a scale from 0-5 based on the presence (1) or absence (0) of the items modified from the Ende's SMART criteria: Specific (S), Measurable (M), Achievable (A), Relevant (R), and Time-bound (T). The primary outcome was any total modified SMART criteria score difference concerning performance level using logistic regression with Generalized Estimating Equations (GEE). Secondary outcomes were differences for individual criteria.

Results

We analyzed 1284 evaluations (311 high performers, 930 expected performers, and 43 below-expected performers) of 94 unique residents from 66 different evaluators. Mean total modified SMART scores were significantly higher in high and below-expected performers than those designated as expected performers by faculty evaluators. Achievable and Relevant written feedback was provided to high performers in a significantly larger proportion than expected and below-expected performers. Only 278 out of 1284 evaluations met criteria for Specific.

Conclusions

Mean total modified SMART feedback scores were significantly greater in high performers and below-expected performers when compared to expected performers. Achievable and Relevant feedback was provided in greater proportions to high performer residents compared to expected and below-expected performers. These findings are a challenge to academic faculty to engage in quality feedback delivery for residents at all performance levels.

Affecting Change: Enhancing Feedback Interactions with Radiology Trainees

Feedback is a critical part of the learning process and is a valuable tool to empower adult learners. Modern feedback theory places the learner at the center of the feedback encounter. Individual and institutional barriers to effective giving and receiving of feedback can be overcome through education and attention to the form and content of feedback. We review the elements of effective feedback and address issues of framing, environmental, and social factors which aid in providing psychological safety and trust, as necessary elements to create a culture of feedback in radiology training programs. We provide practical strategies to empower learners with the necessary skills to solicit, receive, and reflect on feedback.

Improving Formative Feedback in the Operating Room Setting: Developing and Implementing an Initiative to Improve Feedback Quality and Culture

PROBLEM

Formative feedback, given in an ongoing fashion during the learning process, is fundamental to clinical education. However, dissatisfaction with formative feedback among residents is common. Difficulties with formative feedback are intensified in the operating room (OR) setting due to fast pace, space limitations, and frequent rotation of residents and attendings.

APPROACH

In the anesthesiology and critical care department at the University of Pennsylvania Perelman School of Medicine, the authors launched the Feedback Moment initiative from January 2018 to May 2018 in which 24 first-year residents and attendings were given a short series of prompts designed to facilitate regular, high-quality formative feedback. The authors conducted semistructured interviews with residents before and after the initiative to evaluate its impact.

OUTCOMES

In baseline interviews, 18 participating residents stressed the importance of formative feedback but described feeling unsure of their performance due to lack of ongoing constructive input from attendings. They felt hesitant to approach attendings for feedback due to a desire not to interrupt OR workflow or appear incompetent. In follow-up interviews, residents described the initiative as helping to normalize constructive formative feedback but difficult to execute regularly due to OR workflow issues and frequent rotation of attendings with varying approaches.

NEXT STEPS

Challenges faced by participants in this initiative highlight several considerations for effective OR-based formative feedback. Alternative timings for initiating feedback must be considered in light of the hectic nature of the OR workflow. Residents should be equipped with the skills necessary to adapt to varying practice patterns and frequent rotation between attendings, while attendings should be trained to provide a clear rationale for constructive feedback that allows residents to quickly adapt to practice variation. Finally, establishing clear goals among resident-attending pairs is critical to ensuring that formative feedback given in necessarily brief sessions is focused and productive.

System for central venous catheterization training using computer vision-based workflow feedback

OBJECTIVE

To develop a system for training central venous catheterization that does not require an expert observer. We propose a training system that uses video-based workflow recognition and electromagnetic tracking to provide trainees with real-time instruction and feedback.

METHODS

The system provides trainees with prompts about upcoming tasks and visual cues about workflow errors. Most tasks are recognized from a webcam video using a combination of a convolutional neural network and a recurrent neural network. We evaluate the systems ability to recognize tasks in the workflow by computing the percent of tasks that were recognized and the average signed transitional delay between the system and reviewers. We also evaluate the usability of the system using a participant questionnaire.

RESULTS

The system was able to recognize 86.2% of tasks in the workflow. The average signed transitional delay was -0.7 8.7s. The average score on the questionnaire was 4.7 out of 5 for the system overall. The participants found the interactive task list to be the most useful component of the system with an average score of 4.8 out of 5.

CONCLUSION

Overall, the participants were happy with the system and felt that it would improve central venous catheterization training. Our system provides trainees with meaningful instruction and feedback without needing an expert observer to be present.

SIGNIFICANCE

We are able to provide trainees with more opportunities to access instruction and meaningful feedback by using workflow recognition.

The Attending Meritocracy: Implementation of a Novel Team-Based Approach to Provide Effective Resident Feedback

OBJECTIVE

Providing timely quality feedback is an essential responsibility of teaching faculty and is critical for resident assessment and development throughout training. Numerous evaluation platforms have been created to provide immediate and big picture end-of-rotation feedback. Faculty suffer burnout from electronic documentation demands and workload and as a result, evaluation activity is relegated to a lower priority leading to poor compliance. We implemented a novel team-based Attending Meritocracy (AM)¹ program that encompasses monetary, automated reminder, and punitive components, while adding a competition element to further engage faculty. The aim of this study is to determine effectiveness of AM in increasing compliance with resident feedback.

DESIGN, SETTING AND PARTICIPANTS

Surgical faculty (n = 36) were divided into 5 teams according to service and subspecialty. Points could be earned by completing surgical (Firefly, MiniCEX) or rotation (New Innovations) evaluations, leaving comments, and other educational tasks. A prize for the highest scoring team was identified as a dinner financed by the non-winning teams. Data from evaluation platforms was extracted. Continuous variables were compared using Mann-Whitney-U test, and categorical variables using chi-squared test.

RESULTS

When comparing July 2019 to February 2020 (control period) with July 2020 to February 2021(initial implementation period), we found a 237% increase in submitted NI evaluations (n = 111-374) and a 42.5% decrease in median time to completion from 60.4 (33.2-106.9) days to 34.7 (24.0-64.5) days, (p = 0.001).² We observed an increase in operative evaluations completed (Mini CEX n = 4-97, Firefly n = 150-1284).

CONCLUSIONS

Implementation of a team-based attending meritocracy program is an effective budget neutral method to increase completion of resident evaluations. Further investigation is needed to assess improvement in quality of feedback as well as to explore it's impact on progression of resident autonomy.

Formative Feedback with In-Class Question Bank Utilization Improves Resident Satisfaction with General Surgery Didactics

BACKGROUND

Formative feedback provides low-stakes opportunities for educational improvement. To enrich our basic science didactics, formative feedback measures were incorporated into our didactics using mobile devices.

MATERIALS AND METHODS

Lecture changes included institutional paid access to a commercial question bank, a 5-item in-class pre-didactic quiz curated from the question bank and taken on the resident's mobile device, and group discussion of quiz topics. An anonymous survey was sent to participating residents.

RESULTS

Overall response rate was 71% among residents. All reported that the new lecture format was a valuable addition to the basic science curriculum (100% Agree/Strongly Agree), and formative assessments provided valuable feedback about the progress of their learning (Strongly Agree = 42%, Agree =58%). All residents reported that in-class use of their mobile device for quizzes was convenient, with majority (84%) preferring it over paper printouts. Residents were more motivated to study before lecture (Strongly Agree = 42%, Agree =42%), with majority also reporting the new format helped identify weaknesses in their knowledgebase (Strongly Agree = 58%, Agree =33%). While majority of residents agreed that quizzes motivated them to study more after lecture, a large portion disagreed (42%). Majority of senior residents reported that the process of composing quizzes prior to lecture enriched their own learning (57%) and helped them find gaps in their knowledge (71%).

CONCLUSION

Incorporating a commercial question bank within didactics gives general surgery residents formative feedback and encourages learning outside the classroom, leading to improved satisfaction with basic science didactics.

A qualitative assessment of emergency medicine residents' receptivity to feedback

BACKGROUND

Effective feedback is the cornerstone of competency-based education. The emergency department (ED) is a unique learning and feedback environment. Developing our understanding of emergency medicine (EM) residents' experiences around feedback will improve resident training and inform EM faculty development programs.

OBJECTIVE

This qualitative study explores the feedback culture and practices in EM and resident's experiences and attitudes toward feedback in this specific training environment.

METHODS

At a large categorical EM program, 15 residents voluntarily participated in semistructured interviews regarding feedback. These individual interviews were performed by a nonphysician investigator and transcripts underwent an inductive multistep coding process. Transcripts were analyzed to identify common factors influencing feedback and then comparisons were made between residents to explore the interconnectedness of identified factors and further categorize consistent themes.

RESULTS

Factors inherent to the ED environment make the delivery of effective feedback challenging. Residents also revealed that feedback-seeking/-avoidant behavior and receptivity to feedback are multifactorial. Residents actively seek feedback when they feel that they performed well but tend to avoid feedback interactions when they expect constructive feedback. Finally, residents filter feedback based on attending personality and perceived practice style as well as their own desired practice style.

CONCLUSIONS

It is important for program leaders to understand their residents' experiences with feedback and engage both faculty and residents in conversations around feedback delivery and receptivity. An improved understanding of these experiences might also reveal barriers to performance assessment and guide efforts to improve the accuracy and reliability of resident evaluations.

Novel tool for assessing the quality of feedback in the emergency room (FEED-ER)

BACKGROUND

The Accreditation Council for Graduate Medical Education (ACGME) emphasizes constructive feedback as a critical component of residency training. Despite over a decade of using competency-based milestone evaluations, emergency medicine (EM) residency programs lack a standardized method for assessing the quality of feedback. We developed two novel EM-specific feedback surveys to assess the quality of feedback in the ER (FEED-ER) from both the resident and the faculty perspectives. This study aimed to evaluate the surveys' psychometric properties.

METHODS

We developed FEED-ER using a Likert scale with faculty and resident versions based on the ACGME framework and a literature review. The preliminary survey consisted of 25 questions involving the feedback domains of timeliness, respect/communication, specificity, action plan, and feedback culture. We conducted two modified Delphi rounds involving 17 content experts to ensure respondent understanding of the items, item coherence to corresponding feedback domains, thematic saturation of domain content, and time duration. A multicenter study was conducted at five university-based EDs in the United States and one in Thailand in 2019. We evaluated the descriptive statistics of the frequency of responses, validity evidence, and reliability of FEED-ER.

RESULTS

A total of 147 EM faculty and 126 EM residents completed the survey. Internal consistency was adequate (Cronbach's alpha > 0.70) and test-retest reliability showed adequate temporal stability (ICC > 0.80) for all dimensions. Content validity was deemed acceptable (CVC > 0.80) for all items. From the 25 items of FEED-ER, 23 loaded into the originally theorized dimensions (with factor loadings > 0.50). Additionally, the five feedback domains were found to be statistically distinct, with correlations between 0.40 and 0.60. The final survey has 23 items.

CONCLUSIONS

This is the first study to develop and provide validity evidence for an EM-specific feedback tool that has strong psychometric properties, is reproducible and reliable, and provides an objective measure for assessing the quality of feedback in the ED.

Adult Gastroenterology Trainees’ Experience of Receiving Feedback on Their Performance of Endoscopy in the Workplace

Background

Competency-based gastrointestinal endoscopy training is concerned with outcomes of the learning experience. Feedback allows for trainees to achieve the expected outcomes. However, little is known about trainees’ experience of receiving feedback. Gaining understanding of their experience could help improve feedback practices. The study was conducted to explore what it means for adult gastroenterology trainees to receive feedback on their performance of endoscopy in the workplace.

Methods

An interpretative phenomenological approach was used. Individual semi-structured interviews were conducted with six trainees from three Canadian adult gastroenterology residency programs. Interviews were audio-recorded and transcribed verbatim for analysis. Analysis was conducted to identify the phenomenological themes across participants’ accounts of lived experience to provide an insight into the meaning of experiencing the studied phenomenon.

Findings

Three phenomenological themes of experience were identified: taking pauses, negotiating understandings and accepting asymmetry. Taking pauses allowed for participants to receive feedback on their performance of endoscopy. Participants needed to negotiate attending gastroenterologists’ different understandings of gastrointestinal endoscopy while carrying their own whenever feedback was provided. They had to accept the asymmetry between the roles of care provider and learner as well.

Discussion

The study has captured the uniqueness and the complexity of the lived experience of receiving feedback on the performance of endoscopy in the workplace from the perspective of study participants. The gained understanding of this experience has enabled the authors to suggest how attending gastroenterologists’ feedback practices may be improved.

Faculty Self-Evaluation of Experiences with Delivering Feedback to Trainees Across Academic Ranks

BACKGROUND

The ability to provide feedback is a developable faculty skill; however, it is unclear how academic rank impacts experiences with feedback delivery.

METHODS

A survey was distributed to 1258 physicians of all academic ranks at a large academic medical center. Questions explored the respondent's feedback delivery beliefs and barriers.

RESULTS

In total, 96% of respondents agreed feedback is important to resident education. Higher academic rank correlated with increased comfort with feedback delivery, and 89% of respondents experienced at least 1 barrier to feedback delivery.

CONCLUSION

Feedback experiences vary across academic ranks, with full professors being more comfortable with feedback delivery and less likely to experience barriers.

SUPPLEMENTARY INFORMATION

The online version contains supplementary material available at 10.1007/s40670-020-01196-5.

Clinical teachers' motivations for feedback provision in busy emergency departments: a multicentre qualitative study

BACKGROUND

Feedback is an effective pedagogical tool in clinical teaching and learning, but the actual perception by learners of clinical feedback is often described as unsatisfactory. Unlike assessment feedback or teaching sessions, which often happen within protected time and space, clinical feedback is influenced by numerous clinical factors. Little is known about clinical teachers' motivations to provide feedback in busy clinical settings. We aimed to investigate the motivations behind feedback being given in emergency departments (EDs).

METHODS

A qualitative analysis of semi-structured interview data was conducted between August 2015 and June 2016. Eighteen attending physicians were purposively sampled from three teaching hospital EDs in Taiwan. Data were thematically analysed, both inductively (from the data) and deductively (using self-determination theory (SDT)). Themes were mapped to the different motivation types identified by the SDT.

RESULTS AND DISCUSSION

Despite working in busy clinical settings, Taiwanese ED clinical teachers reported being motivated to provide feedback when they felt responsible for their learners, when they understood the importance of feedback (patient safety and partner building), or simply because they were committed to following a tradition of passing on their clinical knowledge to their juniors. Suggestions to facilitate the internalisation of external motivations are proposed.

CONCLUSIONS

In this qualitative study, motivations for clinical feedback were identified. Although the motivations are mostly extrinsic, the elicitation of internal motivation is possible once true satisfaction is fostered during the feedback-giving process. This understanding can be used to develop interventions to enable clinical feedback to be provided in a sustained manner.

Formalizing Feedback: Introducing a Structured Approach in an Outpatient Resident Clinic

OBJECTIVE

Delivering feedback is an integral part of graduate medical education. This paper will present how feedback research informed the development of a new feedback model and discuss its implementation and evaluation by residents in an outpatient psychiatry clinic.

METHOD

After reviewing research, a new feedback model of self-determined goal setting with guided objectives and quarterly formal in-person feedback sessions was implemented with 10 psychiatry residents during their 12-month outpatient experience in postgraduate year (PGY)-3. Residents received a pre-intervention survey to assess existing opinions of feedback and goal setting and a post-intervention survey to evaluate experiences with the new feedback model.

RESULTS

On the pre-intervention survey, 3 of 8 resident respondents indicated they had previously set goals, and only 4 of 8 predicted goal setting would be helpful, with average helpfulness rating of 3.62 (scale of 1 to 5). Cumulatively, 10 PGY-3 residents set 31 goals over the academic year. On the post-intervention survey, resident respondents rated the helpfulness of goal setting at 4.71 and quarterly, formal, in-person feedback meetings at 4.86. Success at reaching their self-determined goals was rated at 5 by all respondents.

CONCLUSIONS

Utilizing self-determined goals and formal in-person feedback sessions seemed to provide the framework for an effective feedback model in an outpatient resident clinic. This pilot project suggests that introducing formal feedback models can have a positive impact on resident clinical and educational growth. The data support expanding the model to assess its generalizability with the goal of furthering development of evidence-based feedback models.

Perceptions of the feedback process: a case study of veterinary residents and supervisors

BACKGROUND

Appraisal of resident clinical performance is critical during training. The most common method for this is feedback to residents based on impressions of supervising clinicians. How effective these practices are for veterinary residents remains unknown. This study focused on establishing perceptions of veterinary residents and supervisors regarding the feedback process.

METHODS

A qualitative case study format was chosen to investigate the perceptions and experiences within a well-developed residency programme. The study cohort consisted of veterinary residents and supervisors from the same specialties. Qualitative data were collected through individual semistructured interviews continuing iteratively until theoretical saturation was reached (14 in total).

RESULTS

Mismatches in resident and supervisor perceptions were evident regarding positive feedback delivery and the importance of dialogue. The nature of the resident/supervisor relationship and the efficacy of feedback were closely interlinked. The development of a feedback-friendly culture would be beneficial. Residents perceived that feedback on teaching was lacking. Milestones were perceived to be lacking.

CONCLUSION

The results highlight a need for change away from the 'no news is good news' culture. Development of training workshops, formation of closer relationships between supervisors and residents, and a transition to competency-based education may be necessary.

Pilot project to assess and improve neurosurgery resident and staff perception of feedback to residents for self-improvement goal formation

OBJECTIVE

The Accreditation Council for Graduate Medical Education (ACGME) has pushed for more frequent and comprehensive feedback for residents during their training, but there is scant evidence for how neurosurgery residents view the current feedback system as it applies to providing information for self-improvement and goal formation. The authors sought to assess neurosurgery resident and staff perceptions of the current resident feedback system in providing specific, meaningful, achievable, realistic, and timely (SMART) goals. The authors then created a pilot project to improve the most unfavorably viewed aspect of the feedback system.

METHODS

The authors conducted an anonymous survey of neurosurgery residents and staff at an academic medical institution to assess SMART goals for resident feedback and used the results to create a pilot intervention to address the most unfavorably viewed aspect of the feedback system. The authors then conducted a postintervention survey to see if perceptions had improved for the target of the intervention.

RESULTS

Neurosurgery residents and staff completed an anonymous online survey, for which the results indicated that resident feedback was not occurring in a timely manner. The authors created a simple anonymous feedback form. The form was distributed monthly to neurosurgery residents, neurosurgical staff, and nurses, and the results were reported monthly to each resident for 6 months. A postintervention survey was then administered, and the results indicated that the opinions of the neurosurgery residents and staff on the timeliness of resident feedback had changed from a negative to a nonnegative opinion (p = 0.01).

CONCLUSIONS

The required ACGME feedback methods may not be providing adequate feedback for goal formation for self-improvement for neurosurgery residents. Simple interventions, such as anonymous feedback questionnaires, can improve neurosurgery resident and staff perception of feedback to residents for self-improvement and goal formation.

An Innovative Feedback Tool Leading to Improved Faculty Feedback and Positive Reception by Residents

Introduction

In 2012 the Accreditation Council for Graduate Medical Education implemented trainee milestones as tools for clinical competency committees to use for evaluation, feedback, remediation, and promotion purposes. Prior to this innovation, there has not been an adequate method to capture, organize, and graphically illustrate the evaluations by attendings in a simple, fast and organized fashion.

Methods

We created a novel, web-based, mobile-friendly evaluation tool to help fill this identified gap. The survey-based program creates a milestone-based evaluation, takes only a few minutes to complete, and easily collates the results in a graphic format creating an individualized “dashboard.” The dashboard is then used by both trainees and their evaluators as a feedback platform.

Results

With the implementation of the dashboard, educational leadership has noted an increase in the number of submitted evaluations of residents and the amount of face-to-face feedback given by attendings to residents. A post-implementation survey of the residents revealed that they found the dashboard-provided feedback more helpful than prior modes of feedback, although the number of evaluations was still too few.

Conclusion

The use of our feedback dashboard is useful to multiple targeted end-users, including general faculty evaluators, program leadership, and the residents themselves for gathering formative feedback that is specific and timely. This tool is adaptable and likely generalizable to other residency programs and specialties.

Clinical learning in the context of uncertainty: a multi-center survey of emergency department residents' and attending physicians' perceptions of clinical feedback

BACKGROUND

Feedback is an essential part of clinical teaching and learning, yet it is often perceived as unsatisfactory in busy clinical settings. Clinical teachers need to balance the competing demands of clinical duty and feedback provision. The influence of the clinical environment and the mutual relationship between feedback giving and seeking has been inadequately investigated. This study therefore aimed to quantify the adequacy, perceptions, and influential factors of feedback provision during resident training in emergency departments (EDs).

METHODS

A multicenter online questionnaire study was undertaken. The respondents comprised ED residents and clinical teachers from four teaching hospitals in Taiwan. The questionnaire was developed via an expert panel, and a pilot study ensured validity. Ninety clinical teachers and 54 residents participated.

RESULTS

The respondents reported that the majority of feedback, which usually lasted 1-5 min, was initiated by the clinical teachers. Feedback satisfaction was significantly lower for the clinical teachers than for the residents (clinical teachers M = 13.8, SD = 1.83; residents M = 15.3, SD = 2.14; p < 0.0001), and positive feedback was provided infrequently in clinical settings (31.1%). Both groups of participants admitted hesitating between providing/seeking feedback and completing clinical work. Being busy, the teachers' clinical abilities, the learners' attitudes, and the relationship between both parties were reported as the most influential factors in feedback provision.

CONCLUSION

ED clinical feedback provision is often short, circumstantial, and initiated by clinical teachers. Providing or seeking feedback appears to be an important part of clinical learning in the context of uncertainty. The importance of the relationship between the feedback seeker and the provider highlights the interactive, reciprocal nature of clinical feedback provision.

Resident Perspective on Feedback and Barriers for Use as an Educational Tool

Background Feedback in physician graduate medical education is not clearly defined. Some parties may view questioning as a form of feedback, others the conversations over lunch, some the comments in the operating room (OR), and still others the written evaluation at planned meetings. The lack of clarity in defining what constitutes feedback is concerning when this is considered a fundamental means of education to enhance practices and care for patients. If residents do not recognize they are receiving feedback, or the response to feedback is met with opposition, then feedback as an educational device can be limited. For this manuscript, feedback is defined as written or verbal comments regarding medical knowledge, performance, technique, or patient care. Objective This study attempts to identify barriers to feedback by identifying attitudes toward feedback processes through a questionnaire. Methods Ten questions were provided to residents at a single institution representing, emergency medicine, family medicine, internal medicine, neurology, and neurosurgery during the 2017-2018 academic year. Response was voluntary and the study was granted exemption by local institutional review board since no identifying information was collected to link responses to specific residents. Questions were formulated to identify how positive or negative a resident felt toward specific aspects of feedback. Results Of the possible 84 resident respondents, 40 residents participated reflecting a response of approximately 48%. Questionnaires revealed that 22.5% of respondents found feedback to be a stressful event. Sixty-seven point five percent (67.5%) of resident respondents associated the prompt that they are about to receive feedback as concerning. Only 2.5% of residents identified a meeting with the program director as a sign that the resident may be doing well. Appointments for feedback were viewed as a positive event in 12.5% of respondents. Ninety-five percent (95%) of residents do not feel that all feedback will affect their permanent record. Ten percent (10%) of residents identified receiving feedback as a positive event. Ninety-five percent (95%) of residents indicated that they have actively tried to change behavior or practices based on feedback. Forty percent (40%) of residents found themselves censoring "negative" feedback. Conclusions Barriers to feedback include the inability to present sensitive subjects in a constructive manner and superficial relationships between the evaluator and resident physician. Research directed at addressing these barriers could lead to improved use of feedback as an educational tool.

Constructing a Shared Mental Model for Feedback Conversations: Faculty Workshop Using Video Vignettes Developed by Residents

INTRODUCTION

Providing feedback is a fundamental principle in medical education; however, as educators, our community lacks the necessary skills to give meaningful, impactful feedback to those under our supervision. By improving our feedback-giving skills, we provide concrete ways for trainees to optimize their performance, ultimately leading to better patient care.

METHODS

In this faculty development workshop, faculty groups used six feedback video vignettes scripted, enacted, and produced by residents to arrive at a shared mental model of feedback. During workshop development, we used qualitative analysis for faculty narratives combined with the findings from a focused literature review to define dimensions of feedback.

RESULTS

Twenty-three faculty (physical medicine and rehabilitation and neurology) participated in seven small-group workshops. Analysis of group discussion notes yielded 343 codes that were collapsed into 25 coding categories. After incorporating the results of a focused literature review, we identified 48 items grouped into 10 dimensions of feedback. Online session evaluation indicated that faculty members liked the workshop's format and thought they were better at providing feedback to residents as a result of the workshop.

DISCUSSION

Small faculty groups were able to develop a shared mental model of dimensions of feedback that was also grounded in medical education literature. The theme of specificity of feedback was prominent and echoed recent medical education research findings. Defining performance expectations for feedback providers in the form of a practical and psychometrically sound rubric can enhance reliable scoring of feedback performance assessments and should be the next step in our work.

Perioperative Teaching and Feedback: How are we doing in Canadian OTL-HNS programs?

Background

Discrepancies between resident and faculty perceptions regarding optimal teaching and feedback during surgery are well known but these differences have not yet been described in Otolaryngology - Head and Neck Surgery (OTL-HNS). The objectives were thus to compare faculty and resident perceptions of perioperative teaching and feedback in OTL-HNS residency programs across Canada with the aim of highlighting potential areas for improvement.

Methods

An anonymous electronic questionnaire was distributed to residents and teaching faculty in OTL-HNS across Canada with additional paper copies distributed at four institutions. Surveys consisted of ratings on a 5-point Likert scale and open-ended questions. Responses among groups were analysed with the Wilcoxon-Mann Whitney test, while thematic analysis was used for the open-ended questions.

Results

A total of 143 teaching faculty and residents responded with statistically significant differences on 11 out of 25 variables. Namely, faculty reported higher rates of pre and intra-operative teaching compared to resident reports. Faculty also felt they gave adequate feedback on residents’ strengths and technical skills contrary to what the residents thought. Both groups did agree however that pre-operative discussion is not consistently done, nor is feedback consistently given or sought.

Conclusion

Faculty and residents in OTL-HNS residency programs disagree on the frequency and optimal timing of peri-operative teaching and feedback. This difference in perception emphasizes the need for a more structured approach to feedback delivery including explicitly stating when feedback is being given, and the overall need for better communication between residents and staff.

Bridging the gap to effective feedback in residency training: perceptions of trainees and teachers

BACKGROUND

Clinical feedback is an important part of residency training, yet literature suggests this complex interaction is not completely understood. In particular, little is known about what resident versus attending physicians expect as feedback. This study investigates this gap in knowledge by examining differences in interactions that residents and attendings view as feedback.

METHODS

Surveys containing sample clinical feedback scenarios were distributed to residents and attending physicians in emergency medicine and general surgery at a large academic medical center. Respondents were asked to decide whether useful feedback was provided in each scenario, and responses were compared between the two groups. Continuous features were summarized with medians, interquartile ranges (IQRs), and ranges; categorical features were summarized with frequency counts and percentages. Comparisons of features between residents and attendings were evaluated using Wilcoxon rank sum, chi-square, and Fisher exact tests. Statistical analyses were performed using version 9.4 of the SAS software package (SAS Institute, Inc.; Cary, NC). All tests were two-sided and p-values < 0.05 were considered statistically significant.

RESULTS

Seventy-two individuals responded to the survey out of approximately 110 invitations sent (65%), including 35 (49%) residents and 37 (51%) attendings. Of 35 residents, 31 indicated their level of training, which included 13 (42%) PGY-1, 9 (29%) PGY-2, 6 (19%) PGY-3, and 3 (10%) PGY-4, respectively. Of 37 attendings, 34 indicated the number of years since completion of residency or last fellowship, at a median of 9 years (IQR 4-14; range 1-31). No significant difference was found in residents' and attendings' perceptions of what constituted feedback in the sample scenarios.

CONCLUSIONS

While this study did not find a statistical difference in perception of feedback between residents and attendings, additional factors should be considered when investigating perceived feedback deficiencies. Further research is needed to better understand and improve the clinical feedback process.

Motivation for feedback-seeking among pediatric residents: a mixed methods study

BACKGROUND

For effective self-directed life-long learning physicians need to engage in feedback-seeking, which means fostering such behavior during training. Self-determination theory (SDT) posits that intrinsic motivation is fostered when the environment optimizes the individual's experience of autonomy, relatedness, and competence. Educational settings meeting these psychological needs may foster intrinsic motivation in trainees, enhance their desire for feedback, and promote feedback-seeking. We sought to examine residents' feedback-seeking behaviors through the lens of SDT and explore the association with intrinsic motivation and career choice.

METHODS

We used a mixed-methods approach with an explanatory sequential design. Residents participated in simulation training, completed an inventory of intrinsic motivation (IMI) and responded to sequential opportunities for performance feedback requiring different levels of effort. We compared IMI scores and career choice between groups with different effort. We interviewed high-effort feedback-seekers and conducted thematic analysis of interview data.

RESULTS

Thirty-four of 35 residents completed the survey (97%). Of those completing the study, 12 engaged in low-effort feedback-seeking only, 10 indicated intent for high-effort feedback-seeking and 10 actually engaged in higher effort to get feedback. Groups did not differ in mean IMI scores. Among high-effort feedback-seekers more residents were interested in critical care-related fields compared to the other groups. We identified 5 themes around autonomy, relatedness, and competence clarifying residents' reasons for feedback-seeking.

CONCLUSIONS

Our findings suggest that among residents, the relationship between motivation and feedback-seeking is complex and cannot be predicted by IMI score. Career plans and relationships with feedback providers impact feedback-seeking, which can inform educational interventions.

An assessment of emotional intelligence in emergency medicine resident physicians

OBJECTIVES

To define the emotional intelligence (EI) profile of emergency medicine (EM) residents, and identify resident EI strengths and weaknesses.

METHODS

First-, second-, and third-year residents (post-graduate years [PGY] 1, 2, and 3, respectively) of Thomas Jefferson University Hospital's EM Program completed the Emotional Quotient Inventory (EQ-i 2.0), a validated instrument offered by Multi-Health Systems. Reported scores included total mean EI, 5 composite scores, and 15 subscales of EI. Scores are reported as means with 95% CIs. The unpaired, two-sample t-test was used to evaluate differences in means.

RESULTS

Thirty-five residents completed the assessment (response rate 97.2%). Scores were normed to the general population (mean 100, SD 15). Total mean EI for the cohort was 103 (95%CI,100-108). EI was higher in female (107) than male (101) residents. PGY-2s demonstrated the lowest mean EI (95) versus PGY-1s (104) and PGY-3s (110). The difference in PGY-3 EI (110; 95%CI,103-116) and PGY-1 EI (95, 95%CI,87-104) was statistically significant (unpaired t-test, p<0.01). Highest composite scores were in interpersonal skills (107; 95%CI,100-108) and stress management (105; 95%CI,101-109). Subscale cohort strengths included self-actualization (107); empathy (107); interpersonal relationships (106); impulse control (106); and stress tolerance (106). Lowest subscale score was in assertiveness (98). Self-regard (89), assertiveness (88), and independence (90) were areas in which PGY-2s attained relatively lower scores (unpaired t-test, p<0.05) compared to their peers and the general population. PGY-3's scored highest in nearly all subscales.

CONCLUSIONS

The EQ-i offers insight into training that may assist in developing EM residents, specifically in self-regard, assertiveness, and self-expression. Further study is required to ascertain if patterns in level of training are idiosyncratic or relate to the natural maturation of residents.

Challenges of feedback provision in the workplace: A qualitative study of emergency medicine residents and teachers

BACKGROUND

Feedback is an effective pedagogical tool in clinical teaching and learning, but is often perceived as unsatisfactory. Little is known about the effect of a busy clinical environment on feedback-giving and -seeking behaviors. This study aims to determine the perceptions and challenges of feedback provision in a busy clinical setting, exemplified by an emergency department (ED).

METHODS

A qualitative semi-structured interview study design was employed. Thirty-six participants (18 attending physicians, 18 residents) were purposively sampled from three EDs in northern Taiwan between August 2015 and July 2016. Interviews were recorded, transcribed, and analyzed thematically.

RESULTS

Three major themes were identified with illustrative quotes: (1) the balance between patient safety and providing feedback, (2) variability in feedback, and (3) influential factors, barriers and enablers.

CONCLUSIONS

In real practice, clinical duties competed with the impulse to provide feedback. The variety and complexity of feedback extended beyond style and content. Clinical and contextual factors - some of which may be presented as barriers - influenced how, when and whether a teacher or learner decided to give or seek feedback.

Development of a curriculum and training program in Woman Veterans Health for Internal Medical Residents

INTRODUCTION

Internal Medicine residents must develop competency as Primary Care Providers, but a gap exists in their curriculum and training with regard to women's reproductive health. With increasing need in VA due to new influx of women veterans it poses problems in recruitment of competent physicians trained in Women's health.

METHODS

An intensive, one-month women's reproductive health curriculum with hands on experience for Internal Medicine residents was provided. Curriculum was taught to the residents who rotated at the Women's Health Clinic for one month. Pre-test and post-test exams were administered. Increase in knowledge of residents in providing gender specific evaluations and management was objectively assessed by changes in post-test scores. Data were analyzed for statistically significant improvement in written tests scores.

RESULTS

Total of 47 Internal Medicine residents rotated through Women's Health Center during the evaluation period. All residents completed both pre-test and post-test exams. The average time to complete the pre-test was 20.5 ± 5.4 min and 19.5 ± 4.8 min for post-test. There was no correlation between the time to complete the pre-test exam and the post-test exam. The total score was significantly improved from 8.5 ± 1.6 to 13.2 ± 1.8 (p < 0.0001).

CONCLUSION

This study shows how to equip physicians in training with information on women's health that enables them to provide safe and gender appropriate care in primary care settings. This practice will reduce the need for frequent referrals for specialized care and thus provide cost saving for patient and health care on the whole.

Beyond the Sandwich: From Feedback to Clinical Coaching for Residents as Teachers

INTRODUCTION

Senior trainees (residents) are poised to be unique effectors of clinical feedback. While several curricula are available to teach residents to give or elicit feedback, our curriculum is unique in that it teaches both the giving and elicitation of feedback and focuses on the longitudinal coaching relationship as opposed to onetime feedback interactions. This curriculum provides a framework, called clinical coaching, for streamlining and enhancing feedback interactions between senior and junior trainees.

METHODS

This curriculum consists of: (1) a video module, (2) an interactive workshop, and (3) role-plays. Participants view the module, which simulates traditional feedback contrasted with the suggested approach. Next, an interactive workshop stimulates reflection on feedback, then defines and demonstrates clinical coaching. Finally, participants practice coaching with prewritten scenarios that illustrate critical steps in clinical coaching.

RESULTS

This workshop was initially conducted in September 2014 with 50 participants. Thirty-nine house staff completed the postcurricular survey (13 had attended the workshop, 26 had not). Recognition of interns soliciting feedback one or more times per week was greater amongst workshop attendees (83% of residents, 78% of interns), as compared to nonattendees (53% of residents, 67% of interns). Preparation to give feedback differed amongst resident attendees versus nonattendees (0% vs. 19%, respectively, reported no preparation).

DISCUSSION

These results highlight a need to increase awareness of and preparedness for the vital role that trainees can play in coaching. Training house staff in coaching has the potential to transform feedback for teachers and learners alike.

Implementation of a Needs-Based, Online Feedback Tool for Anesthesia Residents With Subsequent Mapping of the Feedback to the ACGME Milestones

BACKGROUND

Optimizing feedback that residents receive from faculty is important for learning. The goals of this study were to (1) conduct focus groups of anesthesia residents to define what constitutes optimal feedback; (2) develop, test, and implement a web-based feedback tool; and (3) then map the contents of the written comments collected on the feedback tool to the Accreditation Council for Graduate Medical Education (ACGME) anesthesiology milestones.

METHODS

All 72 anesthesia residents in the program were invited to participate in 1 of 5 focus groups scheduled over a 2-month period. Thirty-seven (51%) participated in the focus groups and completed a written survey on previous feedback experiences. On the basis of the focus group input, an initial online feedback tool was pilot-tested with 20 residents and 62 feedback sessions, and then a final feedback tool was deployed to the entire residency to facilitate the feedback process. The completed feedback written entries were mapped onto the 25 ACGME anesthesiology milestones.

RESULTS

Focus groups revealed 3 major barriers to good feedback: (1) too late such as, for example, at the end of month-long clinical rotations, which was not useful because the feedback was delayed; (2) too general and not specific enough to immediately remedy behavior; and (3) too many in that the large number of evaluations that existed that were unhelpful such as those with unclear behavioral anchors compromised the overall feedback culture. Thirty residents (42% of 72 residents in the program) used the final online feedback tool with 121 feedback sessions with 61 attendings on 15 rotations at 3 hospital sites. The number of feedback tool uses per resident averaged 4.03 (standard deviation 5.08, median 2, range 1-21, 25th-75th % quartile 1-4). Feedback tool uses per faculty averaged 1.98 (standard deviation 3.2, median 1, range 1-25, 25th-75th % quartile 1-2). For the feedback question item "specific learning objective demonstrated well by the resident," this yielded 296 milestone-specific responses. The majority (71.3%) were related to the patient care competency, most commonly the anesthetic plan and conduct (35.8%) and airway management (11.1%) milestones; 10.5% were related to the interpersonal and communication skills competency, most commonly the milestones communication with other professionals (4.4%) or with patients and families (4.4%); and 8.4% were related to the practice-based learning and improvement competency, most commonly self-directed learning (6.1%). For the feedback tool item "specific learning objective that resident may improve," 67.0% were related to patient care, most commonly anesthetic plan and conduct (33.5%) followed by use/interpretation of monitoring and equipment (8.5%) and airway management (8.5%); 10.2% were related to practice-based learning and improvement, most commonly self-directed learning (6.8%); and 9.7% were related to the systems-based practice competency.

CONCLUSIONS

Resident focus groups recommended that feedback be timely and specific and be structured around a tool. A customized online feedback tool was developed and implemented. Mapping of the free-text feedback comments may assist in assessing milestones. Use of the feedback tool was lower than expected, which may indicate that it is just 1 of many implementation steps required for behavioral and culture change to support a learning environment with frequent and useful feedback.

Clinical coaching: Evolving the apprenticeship model for modern housestaff

Feedback is one of the core components of teaching in the clinical setting. Traditionally, this activity has emphasized observations made by senior physicians and delivered to medical trainees. However, the optimal approach to feedback remains uncertain, and the literature abounds with trainee-perceived inadequacies in feedback content, quality, and impact. Moreover, given the multiplicity of demands on trainees and their physician mentors, we propose that medical trainees themselves-specifically, medical residents-are poised to serve as unique adjunct effectors of feedback. We propose a model of "clinical coaching" for residents as teachers, with emphasis on the active roles of both the feedback "giver" and "recipient". We define "clinical coaching" as "a helping longitudinal relationship between coach and apprentice that provides continuing feedback on and assistance with improving performance." Here, "coach" is the more experienced trainee (e.g. supervising resident), and "apprentice" is the less experienced trainee (e.g. intern or medical student). By working to better recognize and prepare residents for this vital role, we propose to encourage efforts to optimize the structure, execution, and impact of feedback in the contemporary climate of medical education.

Disparities in Feedback Provision to Emergency Medical Services Professionals

INTRODUCTION

Feedback to EMS professionals is a critical component for optimizing patient care and outcomes in the prehospital setting. There is a paucity of data concerning the feedback received by prehospital providers.

OBJECTIVES

The objective of this study was to describe the prevalence of feedback received by EMS professionals in the past 30 days including the types, sources, modes, and utility of feedback. The secondary objective was to identify factors associated with receiving any feedback and, specifically, feedback regarding medical care provided.

METHODS

This was a cross-sectional survey examining currently practicing nationally certified EMS patient care providers (EMT or higher) in non-military and non-tribal settings. Data were collected on provider characteristics along with feedback received. Descriptive statistics were calculated, and multivariable logistic regression models were constructed to assess the relationship between EMS provider characteristics and receiving feedback. A non-respondent survey was administered to assess for non-response bias.

RESULTS

Responses from 32,314 EMS providers were received (response rate = 10.4%) with 15,766 meeting inclusion criteria. In the 30 days preceding the survey, 69.4% (n = 10,924) of respondents received at least one type of feedback with 54.7% (n = 8,592) reporting receiving medical care feedback. Multivariable logistic regression modeling indicated that higher certification level, fewer years of experience in EMS, working for a hospital-based agency, air medical service, and higher weekly call volumes were significantly associated with increased odds of having received at least one type of feedback, and specifically medical care feedback. Additionally, providing primarily medical/convalescent transport and more years of EMS experience were significantly associated with decreased odds of receiving feedback.

CONCLUSION

Feedback to EMS providers is critical to improving prehospital care. In this study, nearly a third of providers did not receive any feedback in a 30-day period, and nearly half reported not receiving medical care feedback. Disparities in the frequency of feedback exist between different provider levels and service settings, while reported feedback decreased with years of experience in the profession. Future work is needed to assess the content of feedback and role in improving patient care and safety.

Feedback in Medical Education: A Critical Appraisal

OBJECTIVE

The objective was to review and critically appraise the medical education literature pertaining to feedback and highlight influential papers that inform our current understanding of the role of feedback in medical education.

METHODS

A search of the English language literature in querying Education Resources Information Center (ERIC), PsychINFO, PubMed, and Scopus identified 327 feedback-related papers using either quantitative (hypothesis-testing or observational investigations of educational interventions), qualitative methods (exploring important phenomena in emergency medicine [EM] education), or review methods.Two reviewers independently screened each category of publications using previously established exclusion criteria. Six reviewers then independently scored the remaining 54 publications using a qualitative, quantitative, or review paper scoring system. Each scoring system consisted of nine criteria and used parallel scoring metrics that have been previously used in critical appraisals of education research.

RESULTS

Fifty-four feedback papers (25 quantitative studies, 24 qualitative studies, five review papers) met the a priori criteria for inclusion and were reviewed. Eight quantitative studies, nine qualitative studies, and three review papers were ranked highly by the reviewers and are summarized in this article.

CONCLUSIONS

This inaugural Council of Emergency Medicine Residency Directors Academy critical appraisal highlights 20 feedback in medical education papers that describe the current state of the feedback literature. A summary of current factors that influence feedback effectiveness is discussed, along with practical implications for EM educators and the next steps for research.

Standardized Direct Observation Assessment Tool: Using a Training Video

BACKGROUND

We developed a DVD training tool to educate physicians evaluating emergency residents on accurate Standardized Direct Observation Assessment Tool (SDOT) application.

OBJECTIVE

Our goal was to assess whether this training video improved attendings' and senior residents' SDOT use.

METHODS

Participants voluntarily completed SDOT evaluations based on a scripted "test" video. A DVD with "positive" and "negative" scenarios of proper SDOT use was viewed. It included education on appropriate recording of 26 behaviors. The test scenario was viewed again and follow-up SDOTs submitted. Performances by attendings and residents on the pre- and post-test SDOTs were compared.

RESULTS

Twenty-six attendings and 26 senior residents participated. Prior SDOT experience was noted for 8 attendings and 11 residents. For 20 anchors, participants recorded observed behaviors with statistically significant difference on one each of the pretest (no. 20; p = 0.034) and post-test (no. 14; p = 0.041) SDOTs. On global competency assessments, pretest medical knowledge (p = 0.016) differed significantly between groups. The training intervention changed one anchor (no. 5; p = 0.035) and one global assessment (systems-based practice; p = 0.031) more negatively for residents. Recording SDOTs with exact agreement occurred 48.73% for attendings pretest and 54.41% post-test; resident scores were 45.86% and 49.55%, respectively. DVD exposure slightly raised attending scores (p = 0.289) and significantly lowered resident scores (p = 0.046).

CONCLUSIONS

Exposure to an independently developed SDOT training video tended to raise attending scores, though without significance, while at the same time lowered senior resident scores statistically significantly. Emergency attendings' and senior residents' SDOT scoring rarely differed with significance; about half of anchor behaviors were recorded with exact agreement. This suggests senior residents, with appropriate education, may participate in SDOT assessment.

Do Emergency Medicine Residents Prefer Resident-initiated or Attending-initiated Feedback?

BACKGROUND

Real-time feedback is crucial to improving physician performance. Emerging theory suggests that learner-initiated feedback may be more effective in changing performance than attending-initiated feedback, but little is known about how residents perceive resident- versus attending-initiated feedback.

OBJECTIVES

The primary aim was to determine whether residents' satisfaction varied by learner-versus attending-initiated feedback encounters. We hypothesized that residents would be more satisfied with resident-initiated feedback.

METHODS

This was a multicenter study of five emergency medicine residency programs. We developed a milestones-based, real-time feedback intervention that provided behavioral anchors for ED subcompetencies and prompted a feedback discussion. The intervention was implemented at all sites for a 3-month period from March to November 2014. Residents were asked to initiate one card per shift; attendings were also invited to initiate encounters and, in either instance, asked to provide one specific suggestion for improvement. Residents confidentially rated their satisfaction with feedback on a 10-point scale. Reported satisfaction was categorized as "very satisfied" (score of 10) versus "less than very satisfied" (score < 10). Logistic regression was used to assess the difference in satisfaction between resident- versus attending-initiated feedback, and random effects were used to account for the clustering of repeated ratings within resident and by site.

RESULTS

A total of 785 cards was collected from five sites. Participation varied by site (range = 21-487 cards per site). Of the 587 cards with both feedback initiator and satisfaction data, 67% (396/587) were resident-initiated, and median satisfaction score was 10 (range = 4-10). There was no difference in the odds of being "very satisfied" by resident- vesus attending-initiated encounters (odds ratio = 1.08, 95% confidence interval = 0.41 to 2.83).

CONCLUSIONS

Our results suggest that residents are likely to be as satisfied with self-initiated feedback as attending-initiated feedback. Further research is needed to determine whether resident-initiated feedback is more likely to be incorporated into practice and result in objective performance improvements.

Are All Competencies Equal in the Eyes of Residents? A Multicenter Study of Emergency Medicine Residents' Interest in Feedback

Introduction

Feedback, particularly real-time feedback, is critical to resident education. The emergency medicine (EM) milestones were developed in 2012 to enhance resident assessment, and many programs use them to provide focused resident feedback. The purpose of this study was to evaluate EM residents’ level of interest in receiving real-time feedback on each of the 23 competencies/sub-competencies.

Methods

This was a multicenter cross-sectional study of EM residents. We surveyed participants on their level of interest in receiving real-time on-shift feedback on each of the 23 competencies/sub-competencies. Anonymous paper or computerized surveys were distributed to residents at three four-year training programs and three three-year training programs with a total of 223 resident respondents. Residents rated their level of interest in each milestone on a six-point Likert-type response scale. We calculated average level of interest for each of the 23 sub-competencies, for all 223 respondents and separately by postgraduate year (PGY) levels of training. One-way analyses of variance were performed to determine if there were differences in ratings by level of training.

Results

The overall survey response rate across all institutions was 82%. Emergency stabilization had the highest mean rating (5.47/6), while technology had the lowest rating (3.24/6). However, we observed no differences between levels of training on any of the 23 competencies/sub-competencies.

Conclusion

Residents seem to ascribe much more value in receiving feedback on domains involving high-risk, challenging procedural skills as compared to low-risk technical and communication skills. Further studies are necessary to determine whether residents’ perceived importance of competencies/sub-competencies needs to be considered when developing an assessment or feedback program based on these 23 EM competencies/sub-competencies.

Characteristics of Real-Time, Non-Critical Incident Debriefing Practices in the Emergency Department

INTRODUCTION

Benefits of post-simulation debriefings as an educational and feedback tool have been widely accepted for nearly a decade. Real-time, non-critical incident debriefing is similar to post-simulation debriefing; however, data on its practice in academic emergency departments (ED), is limited. Although tools such as TeamSTEPPS® (Team Strategies and Tools to Enhance Performance and Patient Safety) suggest debriefing after complicated medical situations, they do not teach debriefing skills suited to this purpose. Anecdotal evidence suggests that real-time debriefings (or non-critical incident debriefings) do in fact occur in academic EDs;, however, limited research has been performed on this subject. The objective of this study was to characterize real-time, non-critical incident debriefing practices in emergency medicine (EM).

METHODS

We conducted this multicenter cross-sectional study of EM attendings and residents at four large, high-volume, academic EM residency programs in New York City. Questionnaire design was based on a Delphi panel and pilot testing with expert panel. We sought a convenience sample from a potential pool of approximately 300 physicians across the four sites with the goal of obtaining >100 responses. The survey was sent electronically to the four residency list-serves with a total of six monthly completion reminder emails. We collected all data electronically and anonymously using SurveyMonkey.com; the data were then entered into and analyzed with Microsoft Excel.

RESULTS

The data elucidate various characteristics of current real-time debriefing trends in EM, including its definition, perceived benefits and barriers, as well as the variety of formats of debriefings currently being conducted.

CONCLUSION

This survey regarding the practice of real-time, non-critical incident debriefings in four major academic EM programs within New York City sheds light on three major, pertinent points: 1) real-time, non-critical incident debriefing definitely occurs in academic emergency practice; 2) in general, real-time debriefing is perceived to be of some value with respect to education, systems and performance improvement; 3) although it is practiced by clinicians, most report no formal training in actual debriefing techniques. Further study is needed to clarify actual benefits of real-time/non-critical incident debriefing as well as details on potential pitfalls of this practice and recommendations for best practices for use.

Direct Observation of Clinical Skills Feedback Scale: Development and Validity Evidence

Construct: This article describes the development and validity evidence behind a new rating scale to assess feedback quality in the clinical workplace.

BACKGROUND

Competency-based medical education has mandated a shift to learner-centeredness, authentic observation, and frequent formative assessments with a focus on the delivery of effective feedback. Because feedback has been shown to be of variable quality and effectiveness, an assessment of feedback quality in the workplace is important to ensure we are providing trainees with optimal learning opportunities. The purposes of this project were to develop a rating scale for the quality of verbal feedback in the workplace (the Direct Observation of Clinical Skills Feedback Scale [DOCS-FBS]) and to gather validity evidence for its use.

APPROACH

Two panels of experts (local and national) took part in a nominal group technique to identify features of high-quality feedback. Through multiple iterations and review, 9 features were developed into the DOCS-FBS. Four rater types (residents n = 21, medical students n = 8, faculty n = 12, and educators n = 12) used the DOCS-FBS to rate videotaped feedback encounters of variable quality. The psychometric properties of the scale were determined using a generalizability analysis. Participants also completed a survey to gather data on a 5-point Likert scale to inform the ease of use, clarity, knowledge acquisition, and acceptability of the scale.

RESULTS

Mean video ratings ranged from 1.38 to 2.96 out of 3 and followed the intended pattern suggesting that the tool allowed raters to distinguish between examples of higher and lower quality feedback. There were no significant differences between rater type (range = 2.36-2.49), suggesting that all groups of raters used the tool in the same way. The generalizability coefficients for the scale ranged from 0.97 to 0.99. Item-total correlations were all above 0.80, suggesting some redundancy in items. Participants found the scale easy to use (M = 4.31/5) and clear (M = 4.23/5), and most would recommend its use (M = 4.15/5). Use of DOCS-FBS was acceptable to both trainees (M = 4.34/5) and supervisors (M = 4.22/5).

CONCLUSIONS

The DOCS-FBS can reliably differentiate between feedback encounters of higher and lower quality. The scale has been shown to have excellent internal consistency. We foresee the DOCS-FBS being used as a means to provide objective evidence that faculty development efforts aimed at improving feedback skills can yield results through formal assessment of feedback quality.

Feedback and the educational alliance: examining credibility judgements and their consequences

CONTEXT

Several recent studies have documented the fact that, in considering feedback, learners are actively making credibility judgements about the feedback and its source. Yet few have intentionally explored such judgements to gain a deeper understanding of how the process works or how these judgements might interact to influence engagement with and interpretation of feedback. Using the educational alliance framework, we sought to elaborate an understanding of learners' credibility judgements and their consequences.

METHODS

Using constructivist grounded theory we conducted semi-structured interviews with psychiatry residents. We used a theoretical sampling approach that invited participants with diverse scores based on a previously published feedback survey and an investigator-developed educational alliance inventory. Consistent with the principles of grounded theory analysis, data were collected and analysed in an iterative process to identify themes.

RESULTS

Participants depicted themselves as actively contemplating feedback and considering it thoughtfully in light of complex judgements regarding their supervisor, the relationship with their supervisor and the larger context in which the feedback interactions were occurring. These judgements focused on the supervisor's credibility both as a clinician and as a partner in the educational alliance. The educational alliance is judged by trainees in relation to the supervisor's engagement as an educator, commitment to promoting growth of residents and positive attitude toward them.

CONCLUSIONS

Our findings suggest that credibility is a multifaceted judgement that occurs not only at the moment of the feedback interaction but early in and throughout an educational relationship. It not only affects a learner's engagement with a particular piece of feedback at the moment of delivery, but also has consequences for future engagement with (or avoidance of) further learning interactions with the supervisor. These findings can help medical educators develop a more meaningful understanding of the context in which feedback takes place.

Using Milestones as Evaluation Metrics During an Emergency Medicine Clerkship

BACKGROUND

The Accreditation Council for Graduate Medical Education's (ACGME) Milestones presumes graduating medical students will enter residency proficient at Milestone level 1 for 23 skills. The Next Accreditation System now includes Milestones for each postgraduate specialty, and it is unlikely that schools will document every emergency medicine (EM) applicant's EM-specific skills in their performance evaluation.

OBJECTIVES

The goals of this research were to determine if assessment of the Milestones was feasible during a medical student clerkship and examine the proportion of medical students performing at Milestone level 1.

METHODS

This study was conducted at a center with Liaison Committee on Medical Education-approved medical training and a 4-year EM residency. Using traditional clerkship, we studied the feasibility of an ACGME EM Milestones-based clerkship assessment. Data led to redesign of the clerkship and its evaluation process, including all level 1 anchor(s) to add "occasionally" (>60%), "usually" (>80%), and "always" (100%) on a Likert scale to on-shift assessment forms.

RESULTS

During the feasibility phase (2013-14), 75 students rotated though the clerkship; 55 evaluations were issued and 50 contained the Milestone summary. Eight deficiencies were noted in Milestone 12 and three in Milestone 14. After changes, 49 students rotated under the new evaluation rubric. Of 575 completed on-shift evaluations, 16 Milestone deficiencies were noted. Of 41 institutional evaluations issued, only one student had deficiencies noted, all of which pertained to patient care. All evaluations in this second cohort contained each student's Milestone proficiency.

CONCLUSIONS

Assessment of the Milestones is feasible. Communication of ACGME EM Milestone proficiency may identify students who require early observation or remediation. The majority of students meet the anchors for the Milestones, suggesting that clerkship assessment with the ACGME EM Milestones does not adequately differentiate students.

Resident Perceptions of Giving and Receiving Peer-to-Peer Feedback

BACKGROUND

Peer feedback is increasingly being used by residency programs to provide an added dimension to the assessment process. Studies show that peer feedback is useful, uniquely informative, and reliable compared to other types of assessments. Potential barriers to implementation include insufficient training/preparation, negative consequences for working relationships, and a perceived lack of benefit.

OBJECTIVE

We explored the perceptions of residents involved in peer-to-peer feedback, focusing on factors that influence accuracy, usefulness, and application of the information.

METHODS

Family medicine residents at the University of Michigan who were piloting an online peer assessment tool completed a brief survey to offer researchers insight into the peer feedback process. Focus groups were conducted to explore residents' perceptions that are most likely to affect giving and receiving peer feedback.

RESULTS

Survey responses were provided by 28 of 30 residents (93%). Responses showed that peer feedback provided useful (89%, 25 of 28) and unique (89%, 24 of 27) information, yet only 59% (16 of 27) reported that it benefited their training. Focus group participants included 21 of 29 eligible residents (72%). Approaches to improve residents' ability to give and accept feedback included preparatory training, clearly defined goals, standardization, fewer and more qualitatively oriented encounters, 1-on-1 delivery, immediacy of timing, and cultivation of a feedback culture.

CONCLUSIONS

Residents perceived feedback as important and offered actionable suggestions to enhance accuracy, usefulness, and application of the information shared. The findings can be used to inform residency programs that are interested in creating a meaningful peer feedback process.