Guidelines: the do's, don'ts and don't knows of feedback for clinical education

Managing assessment during curriculum change: Ottawa Consensus Statement

Curriculum change is relatively frequent in health professional education. Formal, planned curriculum review must be conducted periodically to incorporate new knowledge and skills, changing teaching and learning methods or changing roles and expectations of graduates. Unplanned curriculum evolution arguably happens continually, usually taking the form of "minor" changes that in combination over time may produce a substantially different programme. However, reviewing assessment practices is less likely to be a major consideration during curriculum change, overlooking the potential for unintended consequences for learning. This includes potentially undermining or negating the impact of even well-designed and important curriculum changes. Changes to any component of the curriculum "ecosystem "- graduate outcomes, content, delivery or assessment of learning - should trigger an automatic review of the whole ecosystem to maintain constructive alignment. Consideration of potential impact on assessment is essential to support curriculum change. Powerful contextual drivers of a curriculum include national examinations and programme accreditation, so each assessment programme sits within its own external context. Internal drivers are also important, such as adoption of new learning technologies and learning preferences of students and faculty. Achieving optimal and sustainable outcomes from a curriculum review requires strong governance and support, stakeholder engagement, curriculum and assessment expertise and internal quality assurance processes. This consensus paper provides guidance on managing assessment during curriculum change, building on evidence and the contributions of previous consensus papers.

Cultivating Competency in Cardiac Sonography: Aligning Entrustable Professional Activities With Industry Expectations

BACKGROUND & AIM

Echocardiography education involves the teaching and assessment of multiple competencies to ensure work-ready graduates. To connect these competency standards to professional practice, it is important that the industry expectation around specific entrustable professional activities (EPAs) is determined. In Australia, echocardiography examinations are eligible for Medicare reimbursement when performed by sonographers listed on the Australian Sonographers Accreditation Registry (ASAR), either as an Accredited Medical Sonographer or as an Accredited Student Sonographer. A key criterion for acceptance onto the registry is the completion of, or active enrolment in, an accredited cardiac sonography course. Eligible courses apply for accreditation and are assessed by ASAR against their Standards for Accreditation of Sonographer Courses. This study sought to investigate the existing cardiac EPAs and provide insights into the industry's expectations for graduate cardiac sonographers in Australia.

METHODS

Using an anonymous online survey tool, an invitation to participate was circulated via professional sonography groups and social media platforms. Accredited Medical Sonographers, Accredited Student Sonographers or interested stakeholders (academic, employer, medical specialist) working in Australia or New Zealand were invited to complete the survey. Survey questions were structured around the existing EPAs and knowledge items described in published sonography competency documents. Participants were asked if each individual EPA should be considered appropriate at the threshold of graduation, or at a higher level following a period of working in the profession.

RESULTS

There were 211 cardiac sonographers who completed the survey. The majority of respondents (148 of 211, 72.2%) indicated that the current EPAs should be updated. At 80% agreement, the following EPAs were considered essential for the graduate: left ventricular structure and function, right ventricular structure and function, atrial size, valvular disease, systemic hypertension, cardiomyopathies, diseases of the aorta, coronary artery disease, pulmonic hypertension, and basic congenital heart disease. This list is more extensive than the current ASAR-endorsed EPAs, and the findings in this research will guide the revision of current ASAR-endorsed EPAs for graduate-level cardiac sonography.

CONCLUSIONS

The results of this study show Accredited Medical Sonographers completing a cardiac sonography course in Australia should be entrusted to perform a wide range of examinations however, greater alignment between educational providers, ASAR and industry is still required.

'It's more like checking in with an old friend': A qualitative study of medical students' experiences with longitudinal coaches throughout medical school

PURPOSE

Coaching in medical education facilitates learners' growth and development through feedback, goal-setting and support. This study explored how coaching relationships evolve throughout medical school and the impact of longitudinal coaching relationships on medical students' approach to feedback and goal setting in the clinical years.

METHOD

In this qualitative study using a constructivist paradigm, authors purposively sampled 15 senior medical students at University of California, San Francisco, to participate in individual semi-structured interviews (October-November 2021). The authors used an inductive approach to thematic analysis.

RESULTS

The authors identified four themes: First, the student-coach relationship deepened over the course of medical school. Second, students identified factors that sustained and strengthened the student-coach relationship over time: a strong foundation to the relationship, the non-evaluative nature of the relationship, coach supportiveness and responsiveness, and coach knowledge of the institutional landscape. Third, coaches provided individualized advice, assessed trajectory, and guided feedback interpretation. Lastly, students applied skills of soliciting and responding to feedback and creating learning goals, originally learned through coaching experience.

CONCLUSIONS

Coaching relationships, grounded in trust, evolve to meet students' changing needs as they grow into physicians. Students apply feedback and goal-setting skills learned with the coach in clinical settings with other supervisors.

Peer assessment in medical communication skills training in programmatic assessment: A qualitative study examining faculty and student perceptions

INTRODUCTION

Current literature recommends assessment of communication skills in medical education combining different settings and multiple observers. There is still a gap in understanding about whether and how peers assessment facilitates learning in communication skills training.

METHODS

We designed a qualitative study using focus group interviews and thematic analysis, in a medical course in the Netherlands. We aimed to explore medical students' and teachers' experiences, perceptions, and perspectives about challenges and facilitating factors in PACST (Peer assessment in medical communication skills training).

RESULTS

Most of the participants reported that peer feedback was a valuable experience when learning communication skills. The major challenges for the quality and credibility of PACST reported by the participants are the question whether peer feedback is critical enough for learning and the difficulty of actually engaging students in the assessment process.

CONCLUSION

Teachers reviewing students' peer assessments may improve the quality and their credibility and the reviewed assessments can best be used for learning purposes. We suggest to pay sufficient attention to teachers' roles in PACST, ensuring a safe and trustworthy environment and additionally helping students to internalize the value of being vulnerable during the evaluation process.

Feedback-based learning from viewpoints of surgical nursing students: A mixed-method study

BACKGROUND

Feedback-based learning (FBL) focuses on guiding the learning process according to educational objectives and the student's needs. This study aimed to investigate surgical nursing students' perceptions and explore their experiences of FBL.

METHOD

The present study used a mixed-methods sequential explanatory design that was conducted in the quantitative and qualitative phases. Surgical nursing students participated in the quantitative phase (n = 105). In the first phase, students completed two questionnaires about FBL and clinical feedback. Semi-structured face-to-face interviews were used to collect qualitative data in the second phase. Graneheim and Lundman's inductive approaches were used to analyse the qualitative data.

RESULTS

The mean (SD) score for students' perception of FBL was 3.99 ± 0.70. The qualitative results were explored in two themes, "motivational support for improvement" and "unpleasant learning".

CONCLUSION

In this study, the positive and negative aspects of FBL were explained. FBL is perceived as a motivational support mechanism to improve students' capabilities during their academic courses and also prepare them for future careers. Conversely, FBL may experience unpleasant learning due to negative feedback and negative emotions.

Programmatic Assessment of Clinical Reasoning: New Opportunities to Meet an Ongoing Challenge

Issue: Clinical reasoning is essential to physicians' competence, yet assessment of clinical reasoning remains a significant challenge. Clinical reasoning is a complex, evolving, non-linear, context-driven, and content-specific construct which arguably cannot be assessed at one point in time or with a single method. This has posed challenges for educators for many decades, despite significant development of individual assessment methods. Evidence: Programmatic assessment is a systematic assessment approach that is gaining momentum across health professions education. Programmatic assessment, and in particular assessment for learning, is well-suited to address the challenges with clinical reasoning assessment. Several key principles of programmatic assessment are particularly well-aligned with developing a system to assess clinical reasoning: longitudinality, triangulation, use of a mix of assessment methods, proportionality, implementation of intermediate evaluations/reviews with faculty coaches, use of assessment for feedback, and increase in learners' agency. Repeated exposure and measurement are critical to develop a clinical reasoning assessment narrative, thus the assessment approach should optimally be longitudinal, providing multiple opportunities for growth and development. Triangulation provides a lens to assess the multidimensionality and contextuality of clinical reasoning and that of its different, yet related components, using a mix of different assessment methods. Proportionality ensures the richness of information on which to draw conclusions is commensurate with the stakes of the decision. Coaching facilitates the development of a feedback culture and allows to assess growth over time, while enhancing learners' agency. Implications: A programmatic assessment model of clinical reasoning that is developmentally oriented, optimizes learning though feedback and coaching, uses multiple assessment methods, and provides opportunity for meaningful triangulation of data can help address some of the challenges of clinical reasoning assessment.

Is time really of the essence? Timeliness of narrative feedback in ophthalmology CBME assessments

PURPOSE

Competency-based medical education relies on a strong program of assessment, and quality comments play a vital role in ensuring its success. The goal of this study is to determine the effect of the timeliness of assessment completion on the quality of the feedback.

MATERIALS AND METHODS

Using the Quality of Assessment for Learning (QuAL) score 2478 assessments were evaluated. The assessments included those completed between July 2017 and December 2020 for 18 ophthalmology residents. Spearman correlation, Mann-Whitney U and Kruskal-Wallis tests were used to assess variations in QuAL scores based on the timeliness of assessment completion.

RESULTS

The timeliness of assessment completion ranged from 0 to 299 d with the mean time for completion being 3 d. As the delay increased, the QuAL score decreased. Feedback provided 4, 5, and 14 d post-encounter demonstrated statistically significant differences in the QuAL score. Additionally, there was a significant difference in the timeliness of feedback when there is no written comment.

CONCLUSIONS

This study demonstrates that the timeliness of assessment completion might have an effect on the quality of written feedback. Written feedback should be completed within 14 d of the encounter to optimize quantity and quality.

Applying simulation learning theory to identify instructional strategies for Generation Z emergency medicine residency education

Introduction

Generation Z learners are entering emergency medicine (EM) residency training, bringing unique learning preferences that influence their engagement with residency education. To optimally teach and motivate this incoming generation of learners, EM educators must understand and adapt to the changing instructional landscape.

Methodology

The Simulation Leaders Advancing the Next Generation in Emergency Medicine (SLANG-EM) Workgroup was created to identify effective educational strategies for Generation Z learners entering EM. Members were faculty in the Society for Academic Emergency Medicine (SAEM) Simulation Academy, well versed in learning theory supporting simulation-based education (SBE) and actively involved in EM residency education.

Unique treatment/analysis

Through primary and secondary literature searches, the SLANG-EM Workgroup identified four distinctive learning preferences of Generation Z learners: (1) individualized and self-paced learning, (2) engaging and visual learning environments, (3) immediate and actionable feedback, and (4) combined personal and academic support. Workgroup members evaluated these learning preferences using a novel conceptual framework informed by the theoretical principles underpinning SBE, recommending instructional strategies for Generation Z EM residency learners across multiple educational environments.

Implications for educators

Instructional strategies were described for the didactic, simulation, and clinical learning environments. In the didactic environment, identified instructional strategies included meaningful asynchronous education, interactive small-group learning, and improved multimedia design. In the simulation environment, educational innovations particularly suitable for Generation Z learners included learner-centered debriefing, rapid-cycle deliberate practice, and virtual simulation. In the clinical environment, described instructional strategies involved setting learner-centered goals and delivering facilitative feedback in the context of an educational alliance. Overall, these instructional strategies were clustered around themes of student-centered education and the educator as facilitator, which align well with Generation Z learning preferences. These findings were synthesized and presented as an advanced workshop, "Delivering Effective Education to the Next Generation," at the 2023 SAEM Annual Meeting.

Designing feedback processes in the workplace-based learning of undergraduate health professions education: a scoping review

BACKGROUND

Feedback processes are crucial for learning, guiding improvement, and enhancing performance. In workplace-based learning settings, diverse teaching and assessment activities are advocated to be designed and implemented, generating feedback that students use, with proper guidance, to close the gap between current and desired performance levels. Since productive feedback processes rely on observed information regarding a student's performance, it is imperative to establish structured feedback activities within undergraduate workplace-based learning settings. However, these settings are characterized by their unpredictable nature, which can either promote learning or present challenges in offering structured learning opportunities for students. This scoping review maps literature on how feedback processes are organised in undergraduate clinical workplace-based learning settings, providing insight into the design and use of feedback.

METHODS

A scoping review was conducted. Studies were identified from seven databases and ten relevant journals in medical education. The screening process was performed independently in duplicate with the support of the StArt program. Data were organized in a data chart and analyzed using thematic analysis. The feedback loop with a sociocultural perspective was used as a theoretical framework.

RESULTS

The search yielded 4,877 papers, and 61 were included in the review. Two themes were identified in the qualitative analysis: (1) The organization of the feedback processes in workplace-based learning settings, and (2) Sociocultural factors influencing the organization of feedback processes. The literature describes multiple teaching and assessment activities that generate feedback information. Most papers described experiences and perceptions of diverse teaching and assessment feedback activities. Few studies described how feedback processes improve performance. Sociocultural factors such as establishing a feedback culture, enabling stable and trustworthy relationships, and enhancing student feedback agency are crucial for productive feedback processes.

CONCLUSIONS

This review identified concrete ideas regarding how feedback could be organized within the clinical workplace to promote feedback processes. The feedback encounter should be organized to allow follow-up of the feedback, i.e., working on required learning and performance goals at the next occasion. The educational programs should design feedback processes by appropriately planning subsequent tasks and activities. More insight is needed in designing a full-loop feedback process, in which specific attention is needed in effective feedforward practices.

Becoming confidently competent: a qualitative investigation of training in cognitive functional therapy for persistent low back pain

BACKGROUND

Physiotherapists trained to deliver biopsychosocial interventions for complex musculoskeletal pain problems often report difficulties in confidence and competency at the end of training. Cognitive Functional Therapy (CFT) is an individualized biopsychosocial intervention and understanding the facilitators and barriers to training in CFT will help inform future training programs. This study aimed to explore physiotherapists' and trainers' perceptions of the process of developing competency in CFT.

METHODS

A cross-sectional qualitative design using interviews of 18 physiotherapists and two trainers investigated training in CFT for persistent LBP via reflexive thematic analysis.

RESULTS

Physiotherapists reported undergoing a complex behavior change process during training. Four themes emerged: 1) Pre-training factors; 2) Behavior change process; 3) Physiotherapy culture and context; and 4) Confident competence and beyond. Key components included graduated practice exposure linked to experiential learning with feedback and clear competency guidelines. Pre-training and contextual factors were facilitators or barriers depending on the individual. Physiotherapists supported ongoing learning, even after competency was achieved.

CONCLUSIONS

This study provides insight into the processes of change during progress toward competency in CFT. It highlights facilitators and barriers to competency including physiotherapy culture and the clinical environment. The study also describes important educational components, including experiential learning and clinical integration, which may be used to inform future post-graduate training.

Evidence-based simulation: Fostering competency through structured and multisource feedback

ABSTRACT

Preclinical simulation is an evidence-based method for nurse practitioner (NP) students to improve clinical communication and disease management competency. During simulation, students may receive feedback from multiple sources, including standardized patients (SPs), faculty, peers, and themselves. Although evidence supports simulation with multisource feedback, its impact on clinical knowledge and communication has yet to be evaluated among NP students. We designed, implemented, and evaluated a preclinical simulation program with structured multisource feedback integrated into a disease management course within a Doctor of Nursing Practice curriculum. Differences in communication self-efficacy and disease management knowledge before and after participation, as well as perceptions of learning and importance of varying feedback sources, were evaluated using a single group pre-post mixed-methods design. On average, clinical communication self-efficacy was significantly higher, and disease management knowledge scores were significantly higher after participation. Learners rated feedback sources as important or very important and described varying feedback sources as complementary. Feedback from SPs, peers, learners themselves, and faculty was complementary and important to learning. This preclinical simulation program with purposeful integration of multisource feedback provides an evidence-based foundation for scaffolding multidomain competency development into curriculums to meet updated standards of advanced nursing education.

The 'Kidney' model for optimising feedback in undergraduate clinical communication: A meta-ethnographic systematic review

OBJECTIVES

Feedback frameworks/models focus on certain aspects of the feedback process, but a coherent and systematic model is lacking. A meta-ethnography was conducted to identify and synthesise guidance for optimising feedback interactions in undergraduate clinical communication simulations.

METHODS

A systematic search of 4 electronic databases and grey literature was conducted. Following Noblit and Hare's seven phases for conducting meta-ethnography, key themes and concepts were synthesised to provide new interpretations of components in effective feedback interactions.

RESULTS

373 publications were identified and 14 included for the final synthesis, which informed the development of a new Feedback Kidney Model. The Model illustrates the interconnections of various components that allow for effective feedback interactions. The main processes include preparation, proactivity, analysis and feedback information, reception and response, and influencing factors.

CONCLUSIONS

This meta-ethnography moves beyond providing an up-to-date synthesis of feedback guidance to proposing the brand-new Feedback Kidney Model, which can guide medical education and future research into how feedback is co-constructed and utilised to promote learning.

PRACTICE IMPLICATIONS

Clinical communication should incorporate meta-cognitive training and using this Model will help students better utilise on-site face-to-face feedback to enhance their learning and improve future communication with patients.

Educating correct ECG interpretation is a way to promote the satisfaction and competency the diagnostic service providers

Background

Given electrocardiogram (ECG) interpretation as one of the diagnostical challenges for medical students and health professionals, this research was carried out to present an experience of web-based teaching method and novel approaches used for training of ECG interpretation.

Methods

This online program was conducted in three days. The main content of the class was taught during one hour, and after that, the teacher spent enough time for responding the asked questions. The components of a normal ECG and different changes that can occur in these waves were taught through clinical case-based scenarios using the web platform and Adobe Connect software. The participants' satisfaction was assessed with a 12-item questionnaire, and the short-term retention of ECG interpretation skill was examined by comparing the posttest scores with pretest.

Results

A total of 224 individuals completed the course. Total satisfaction score was 53.05±6.98 (out of the maximum score of 60). Based on the results of the paired t test, the interpretation skill scores of the participants increased significantly from 2.5 ± 1.57 to 6.96 ± 1.89. (p<0.001, CI = -4.8 to- 4.11).

Conclusion

This web-based nationwide training program provided a supplementary resource for ECG learning among medical students and health-care providers.

Automating the Identification of Feedback Quality Criteria and the CanMEDS Roles in Written Feedback Comments Using Natural Language Processing

Introduction

Manually analysing the quality of large amounts of written feedback comments is time-consuming and demands extensive resources and human effort. Therefore, this study aimed to explore whether a state-of-the-art large language model (LLM) could be fine-tuned to identify the presence of four literature-derived feedback quality criteria (performance, judgment, elaboration and improvement) and the seven CanMEDS roles (Medical Expert, Communicator, Collaborator, Leader, Health Advocate, Scholar and Professional) in written feedback comments.

Methods

A set of 2,349 labelled feedback comments of five healthcare educational programs in Flanders (Belgium) (specialistic medicine, general practice, midwifery, speech therapy and occupational therapy) was split into 12,452 sentences to create two datasets for the machine learning analysis. The Dutch BERT models BERTje and RobBERT were used to train four multiclass-multilabel classification models: two to identify the four feedback quality criteria and two to identify the seven CanMEDS roles.

Results

The classification models trained with BERTje and RobBERT to predict the presence of the four feedback quality criteria attained macro average F1-scores of 0.73 and 0.76, respectively. The F1-score of the model predicting the presence of the CanMEDS roles trained with BERTje was 0.71 and 0.72 with RobBERT.

Discussion

The results showed that a state-of-the-art LLM is able to identify the presence of the four feedback quality criteria and the CanMEDS roles in written feedback comments. This implies that the quality analysis of written feedback comments can be automated using an LLM, leading to savings of time and resources.

Adaptation and validation of the encoding of observations using CONSUL-MCC: A self-determination theory-based tool to observe consultations in maternity care

Objective

During autonomy-supportive consultations, professionals use a need-supportive interaction style to facilitate patients' self-regulated behaviour. To improve maternity care professionals' need-supportive interactions, it is important to provide insights into their interaction style. No tool is currently available for measuring need-supportive interactions in maternity care. Therefore, the aim of this study was to adapt the COUNSEL-CCE to evaluate need-supportive interactions in maternity care and to validate their measurements.

Methods

A five-step adaptation and validation process was performed based on the guideline of Sousa and Rojjanasrirat: 1) adaptation of COUNSEL-CCE by two authors independently; 2) development of a consensus-based tool: CONSUL-MCC; 3) qualitative assessment of CONSUL-MCC; 4) pilot testing of CONSUL-MCC in the target population (N = 10) and 5) psychometric testing in the target population (N = 453).

Results

All indicators of the original tool remained relevant. Four items were rephrased, one indicator was added, and all examples were adapted to maternity care. The results of psychometric testing indicated good construct validity. However, the data characteristics made it impossible to prove the presumed factor structure and perform an accurate intraclass correlation.

Conclusions and innovation

COUNSEL-CCE proved to be a new instrument to gain insights into professionals' interactions and be applied to maternity care.

Preceptor Tools for Success: Student Review on Creating an Effective Clinical Rotation

PURPOSE

The purpose of this article was to explore student perspectives regarding effective preceptors with the intention to improve development of supervised clinical practice experiences.

METHODS

In total, 136 clinical year physician assistant (PA) students, representing 9 accredited PA programs, participated in a mixed-methods survey, including a 5-point Likert scale to identify characteristics of an effective preceptor. The survey achieved a 36.2% response rate. Quantitative data were analyzed based on mean and standard deviation (SD) with the analysis of variance to promote generalizability. Thematic analysis was performed on qualitative responses.

RESULTS

Students reported the highest level of agreement that an effective preceptor welcomes questions from the student (mean 4.88, SD 0.54), are enthusiastic about having a student (4.84, 0.52), and provide a welcoming support staff who encourages student education (4.71, 0.66). Students also highlighted the positive effect of preceptor engagement during clinical rotations.

CONCLUSION

A preceptor who actively engages their student has the greatest positive impact on clinical education. Preceptors can implement clinical teaching strategies from this article to improve student-perceived effectiveness of their rotation.

Documenting Performance and Feedback in Medical Education: An Essential Skill

Documentation of performance provides feedback to medical trainees on their progress and is required by Programs as a record for monitoring whether trainees are achieving expectations against accepted standards. Despite the importance of performance documentation and the focus on improving feedback delivery in medical education, there has been little written in the literature on documentation best practices. Documentation is an essential skill that cannot be learned solely by observation, as most is confidential. However, teaching documentation of learner performance is rarely discussed. This article describes the authors' experience in teaching the skill of effective documentation of trainee performance for the purpose of providing feedback, monitoring progress, and recording evaluations.

When students' words hurt: 12 tips for helping faculty receive and respond constructively to student evaluations of teaching

Student evaluations of curricular experiences and instructors are employed by institutions to obtain feedback and guide improvement. However, to be effective, evaluations must prompt faculty action. Unfortunately, evaluative comments that engender strong reactions may undermine the process by hindering innovation and improvement steps. The literature suggests that faculty interpret evaluation feedback as a judgment not just on their teaching ability but on their personal and professional identity. In this context, critical evaluations, even when constructively worded, can result in disappointment, hurt, and shame. The COVID pandemic has challenged institutions and faculty to repeatedly adapt curricula and educational practices, heightening concerns for faculty burnout. In this context, the risk of 'words that hurt' is higher than ever. This article offers guidance for faculty and institutions to support effective responses to critical feedback and ameliorate counterproductive effects of learner evaluations.

An overview of current evaluation methods of postgraduate dental specialty residents

PURPOSE

A lack of contemporary dental literature exists on evaluating dental residents in postgraduate education, with no standardized criteria or information on technology adoption. This study aims to understand current evaluation methods in dental residency programs and compare them to existing literature.

METHODS

A survey with 22 questions was sent to program directors of 749 ADA/CODA (American Dental Association/Commission on Dental Accreditation)-accredited postgraduate dental residencies. The questions focused on evaluation frequency, faculty involvement, submission methods, and electronic software usage.

RESULTS

The survey response rate was 30.2% (226 responses). Results show that 98% of program directors are involved in evaluations, but require more support from auxiliary faculty. Evaluations are typically submitted twice or four times a year, with 12% of programs wanting increased frequency. Face-to-face delivery of feedback is considered crucial. While desktop computers are widely used for evaluation submission, 55% of programs utilize mobile devices, which correlates with higher program director satisfaction. The most popular evaluation software includes New Innovations, MEd Hub, and Qualtrics. Overall, 86.96% of program directors are satisfied with current evaluation practices. Areas needing improvement are non-program director faculty involvement, resident response/feedback, and software navigation.

CONCLUSION

This study found that a majority of program directors of the postgraduate dental education system are using electronic technology for their resident evaluation systems and are satisfied with their current mechanism of evaluation.

Student mistakes and teacher reactions in bedside teaching

We analyse interactions between teachers and students during video-recorded bedside teaching sessions in internal medicine, orthopaedics and neurology. Multiple raters used a high-inference categorical scheme on 36 sessions. Our research questions concern the types of student mistakes, clinical teachers' reactions to them and if they use different strategies to address different types of mistakes. We used a Poisson model and generalized mixed models to analyse these research questions. Most frequently, students made reproduction mistakes. Relatively high rates of rejection and a similar prevalence of low and high levels of elaboration and correction time for students were observed. Reproduction mistakes were associated with the highest level of rejection and the lowest level of elaboration. High levels of elaboration were observed when students were applying skills in new situations. Students were most often allowed time to correct when mistakes in the areas of analysis or application of skills and knowledge had occurred. There is a decrease in the rate of making mistakes for neurology and orthopaedics compared to internal medicine. Reproduction mistakes influence significantly the outcome feedback compared to application mistakes. Analytic and reproduction mistakes influence elaboration significantly compared to application mistakes. We found a significant effect whether the lecturer allows time for correction of reproduction mistakes compared to application mistakes. These results contribute to the understanding of interactive, patient-centred clinical teaching as well as student mistakes and how teachers are reacting to them. Our descriptive findings provide an empirical basis for clinical teachers to react to student mistakes in didactically fruitful ways.

Twelve tips for practical clinical skills coaching

Coaching is rapidly evolving in clinical medicine, including for clinical skills (CS) learning. Yet a schema is needed for how to coach students in the many CS that are pivotal to the practice of medicine. These twelve tips aim to provide practical strategies for teachers and educators to coach students for CS learning. The tips cover many important aspects of CS coaching, including establishing a safe space, ways to prepare to coach, setting goals, guiding a coaching relationship, fostering coaching conversations, and in-person or virtual approaches. Together, the tips align as seven key steps of an overall coaching process. The twelve tips apply equally to coaching struggling students and all students seeking to improve CS and offer a guide for coaching at an individual or program level.

Developing Evidence for Equitable Assessment Characteristics Based on Clinical Learner Preferences Using Discrete Choice Experiments

PURPOSE

Medical education is only beginning to explore the factors that contribute to equitable assessment in clinical settings. Increasing knowledge about equitable assessment ensures a quality medical education experience that produces an excellent, diverse physician workforce equipped to address the health care disparities facing patients and communities. Through the lens of the Anti-Deficit Achievement framework, the authors aimed to obtain evidence for a model for equitable assessment in clinical training.

METHOD

A discrete choice experiment approach was used which included an instrument with 6 attributes each at 2 levels to reveal learner preferences for the inclusion of each attribute in equitable assessment. Self-identified underrepresented in medicine (UIM) and not underrepresented in medicine (non-UIM) (N = 306) fourth-year medical students and senior residents in medicine, pediatrics, and surgery at 9 institutions across the United States completed the instrument. A mixed-effects logit model was used to determine attributes learners valued most.

RESULTS

Participants valued the inclusion of all assessment attributes provided except for peer comparison. The most valued attribute of an equitable assessment was how learner identity, background, and trajectory were appreciated by clinical supervisors. The next most valued attributes were assessment of growth, supervisor bias training, narrative assessments, and assessment of learner's patient care, with participants willing to trade off any of the attributes to get several others. There were no significant differences in value placed on assessment attributes between UIM and non-UIM learners. Residents valued clinical supervisors valuing learner identity, background, and trajectory and clinical supervisor bias training more so than medical students.

CONCLUSIONS

This study offers support for the components of an antideficit-focused model for equity in assessment and informs efforts to promote UIM learner success and guide equity, diversity, and inclusion initiatives in medical education.

Feedback that Lands: Exploring How Residents Receive and Judge Feedback During Entrustable Professional Activities

Introduction

Receiving feedback from different types of assessors (e.g., senior residents, staff supervisors) may impact trainees' perceptions of the quantity and quality of data during entrustable professional activity (EPA) assessments. We evaluated the quality of EPA feedback provided by different assessors (senior residents, chief medical residents/subspecialty residents, and staff) and explored residents' judgements of the value of this feedback.

Methods

From a database of 2228 EPAs, we calculated the frequency of contribution from three assessor groups. We appraised the quality of 60 procedure-related EPAs completed between July 2019 and March 2020 using a modified Completed Clinical Evaluation Report Rating (CCERR) tool. Next, we asked 15 internal medicine residents to sort randomly selected EPAs according to their judgements of value, as an elicitation exercise before a semi-structured interview. Interviews explored participants' perceptions of quality of written feedback and helpful assessors.

Results

Residents completed over 60% of EPA assessments. We found no difference in modified-CCERR scores between the three groups. When judging EPA feedback value, residents described a process of weighted deliberation, considering perceived assessor characteristics (e.g., credibility, experience with EPA system), actionable written comments, and their own self-assessment.

Discussion

Like other recent studies, we found that residents contributed most to procedure-related EPA assessments. To the established list of factors influencing residents' judgements of feedback value, we add assessors' adherence to, and their shared experiences of being assessed within, EPA assessment systems. We focus on the implications for how assessors and leaders can build credibility in themselves and in the practices of EPA assessments.

Development and Implementation of a Quick Response (QR) Code System to Streamline the Process for Fellows' Evaluation in the Pediatric Intensive Care Unit (PICU) and the Neonatal Intensive Care Unit (NICU) at a Large Academic Center

BACKGROUND/OBJECTIVE

Useful feedback and evaluation are critical to a medical trainee's development. While most academic physicians understand that giving feedback to learners is essential, many do not consider the components of feedback to be truly useful, and there are barriers to implementation. We sought to use a quick reader (QR) system to solicit feedback for trainees in two pediatric subspecialties (pediatric critical care and neonatal-perinatal medicine) at one institution to increase the quality and quantity of feedback received.

METHODS

New valuations were modified from the existing evaluations and imported into online systems with QR code capability. Each fellow was given a QR code linking to evaluations and encouraged to solicit feedback and evaluations in a variety of clinical settings and scenarios. Evaluation numbers and quality of evaluations were assessed and compared both pre- and post-intervention.

RESULTS

There were increases in the number of evaluations completed for both the pediatric critical care fellows and the neonatal-perinatal medicine fellows. There was no overall change in the quality of written evaluations received. Satisfaction with the evaluation system improved for both faculty and fellows of both training programs.

CONCLUSION

In our critical care units, we were successfully able to implement a QR code-driven evaluation for our fellows that improved access for the faculty and offered the ability of the learner to solicit evaluations, without compromising the number or quality of evaluations. What's new: Quick reader (QR) codes can be used by learners to solicit evaluations and feedback from faculty. They can increase the quantity of written evaluations received without affecting their quality.

Feedback conversations: First things first?

OBJECTIVE

This study explores medical students' perceptions regarding the order in which feedback is given and its impact on how that feedback is received.

METHODS

Medical students were interviewed regarding their feedback experiences during medical school and preferred order in which to receive feedback. Thematic analysis was applied to interview transcripts to identify salient themes in students' comments related to feedback order.

RESULTS

Twenty-five students entering their second, third, and fourth years of medical school participated in the study. Students indicated that the order in which feedback was conveyed influenced their receptivity to its content, but varied in their specific order preferences. Most students indicated that they preferred feedback conversations that started with positive observations. Only the most senior students expressed a preference for feedback based on self-assessment.

CONCLUSION

Feedback conversations are complicated interactions. Students' responses to feedback are influenced by a variety of factors, including the order in which feedback is delivered.

PRACTICE IMPLICATIONS

Educators should recognize that students' feedback needs may be influenced by a variety of factors, and should aim to tailor feedback and the order of its delivery to the learner.

Twelve tips for optimising learning for postgraduate doctors in the operating theatre

Learning in the operating theatre forms a critical part of postgraduate medical education. Postgraduate doctors present a diverse cohort of learners with a wide range of learning needs that will vary by their level of experience and curriculum requirements. With evidence of both trainee dissatisfaction with the theatre learning experience and reduced time spent in the operating theatre, which has been exacerbated by the effects of the Covid-19 pandemic, it is vital that every visit to the operating theatre is used as a learning opportunity. We have devised 12 tips aimed at both learners and surgeons to optimise learning in the operating theatre, set out into four domains: educational context, preparation, learning in theatre, feedback and reflection. These tips have been created by a process of literature review and acknowledgment of established learning theory, with further discussion amongst surgical trainees, senior surgical faculty, surgical educators and medical education faculty.

The integrative feedback tool: assessing a novel feedback tool among emergency medicine residents

OBJECTIVE

Feedback is critical to the growth of learners. However, feedback quality can be variable in practice. Most feedback tools are generic, with few targeting emergency medicine. We created a feedback tool designed for emergency medicine residents, and this study aimed to evaluate the effectiveness of this tool.

METHODS

This was a single-center, prospective cohort study comparing feedback quality before and after introducing a novel feedback tool. Residents and faculty completed a survey after each shift assessing feedback quality, feedback time, and the number of feedback episodes. Feedback quality was assessed using a composite score from seven questions, which were each scored 1 to 5 points (minimum total score, 7 points; maximum, 35 points). Preintervention and postintervention data were analyzed using a mixed-effects model that took into account the correlation of random effects between study participants.

RESULTS

Residents completed 182 surveys and faculty members completed 158 surveys. The use of the tool was associated with improved consistency in the summative score of effective feedback attributes as assessed by residents (P=0.040) but not by faculty (P=0.259). However, most of the individual scores for attributes of good feedback did not reach statistical significance. With the tool, residents perceived that faculty spent more time providing feedback (P=0.040) and that the delivery of feedback was more ongoing throughout the shift (P=0.020). Faculty felt that the tool allowed for more ongoing feedback (P=0.002), with no perceived increase in the time spent delivering feedback (P=0.833).

CONCLUSION

The use of a dedicated tool may help educators provide more meaningful and frequent feedback without impacting the perceived required time needed to provide feedback.

Combining Support and Assessment in Health Professions Education: Mentors' and Mentees' Experiences in a Programmatic Assessment Context

Introduction

Mentors in programmatic assessment support mentees with low-stakes feedback, which often also serves as input for high-stakes decision making. That process potentially causes tensions in the mentor-mentee relationship. This study explored how undergraduate mentors and mentees in health professions education experience combining developmental support and assessment, and what this means for their relationship.

Methods

The authors chose a pragmatic qualitative research approach and conducted semi-structured vignette-based interviews with 24 mentors and 11 mentees that included learners from medicine and the biomedical sciences. Data were analyzed thematically.

Results

How participants combined developmental support and assessment varied. In some mentor-mentee relationships it worked well, in others it caused tensions. Tensions were also created by unintended consequences of design decisions at the program level. Dimensions impacted by experienced tensions were: relationship quality, dependence, trust, and nature and focus of mentoring conversations. Mentors and mentees mentioned applying various strategies to alleviate tensions: transparency and expectation management, distinguishing between developmental support and assessment, and justifying assessment responsibility.

Discussion

Combining the responsibility for developmental support and assessment within an individual worked well in some mentor-mentee relationships, but caused tensions in others. On the program level, clear decisions should be made regarding the design of programmatic assessment: what is the program of assessment and how are responsibilities divided between all involved? If tensions arise, mentors and mentees can try to alleviate these, but continuous mutual calibration of expectations between mentors and mentees remains of key importance.

Exploring the Roles and Responsibilities of an Educational Supervisor in Medical Training Within the United Kingdom

Medical trainees or junior doctors within the United Kingdom, regardless of their level of training or specialty, are assigned an educational supervisor (ES). The General Medical Council within the United Kingdom defines an ES as "a trainer who is selected and appropriately trained to be responsible for the overall supervision and management of a specified trainee's educational progress during a clinical placement or series of placements." This article critically evaluates the current literature to explore the roles of the ES in supporting and monitoring a trainee's progress while discussing challenges associated with the role. Through clearer delineation of the role of an ES, barriers to improving training can be identified and overcome, thus improving overall satisfaction with training.

Exploring the Impact of Entrustable Professional Activities on Feedback Culture: A Qualitative Study of Anesthesiology Residents and Attendings

PURPOSE

Entrustable professional activities (EPAs) were introduced as a potential way to optimize workplace-based assessments. Yet, recent studies suggest that EPAs have not yet overcome all of the challenges to implementing meaningful feedback. The aim of this study was to explore the extent to which the introduction of EPAs via mobile app impacts feedback culture as experienced by anesthesiology residents and attending physicians.

METHOD

Using a constructivist grounded theory approach, the authors interviewed a purposive and theoretical sample of residents (n = 11) and attendings (n = 11) at the Institute of Anaesthesiology, University Hospital of Zurich, where EPAs had recently been implemented. Interviews took place between February and December 2021. Data collection and analysis were conducted iteratively. The authors used open, axial, and selective coding to gain knowledge and understanding on the interplay of EPAs and feedback culture.

RESULTS

Participants reflected on a number of changes in their day-to-day experience of feedback culture with the implementation of EPAs. Three main mechanisms were instrumental in this process: lowering the feedback threshold, change in feedback focus, and gamification. Participants felt a lower threshold to feedback seeking and giving and that the frequency of feedback conversations increased and tended to be more focused on a specific topic and shorter, while feedback content tended to focus more on technical skills and more attention was given to average performances. Residents indicated that the app-based approach fostered a game-like motivation to "climb levels," while attendings did not perceive a game-like experience.

CONCLUSIONS

EPAs may offer a solution to problems of infrequent occurrence of feedback and invite attention to average performances and technical competencies, but may come at the expense of feedback on nontechnical skills. This study suggests that feedback culture and feedback instruments have a mutually interacting influence on each other.

What does "Timely" Mean to Residents? Challenging Feedback Assumptions in Postgraduate Education

Introduction

Current orthodoxy states that feedback should be timely and face-to-face, yet the optimal timing and mode of delivery for feedback is unclear. We explored what "optimal timing" means from residents' points of view as feedback providers and receivers, to ultimately inform strategies to optimize feedback in training.

Methods

As near-peers who have dual roles in both providing and receiving feedback, 16 subspecialty (PGY4 and 5) internal medicine residents were interviewed about their perceptions of the optimal timing and format of feedback. Using constructivist grounded theory, interviews were conducted and analyzed iteratively.

Results

Drawing on their experiences as providers and recipients, residents described simultaneously considering and weighing multiple factors when deciding on when and how to provide feedback. These included their own readiness to engage in providing meaningful feedback, the perceived receptiveness of the learner and the apparent urgency of feedback delivery (e.g., if patient safety was at stake). Face-to-face verbal feedback was valued for encouraging dialogue but could be uncomfortable and limited by time constraints. Written feedback could be more honest and concise, and the possibility of asynchronous delivery had potential to overcome issues with timing and discomfort.

Discussion

Participants' perceptions of the optimal timing of feedback challenge current assumptions about the benefits of "immediate" versus "delayed". The concept of "optimal timing" for feedback was found to be complex and context-dependent, defying a formulaic approach. There may be a role for asynchronous and/or written feedback, which has potential to address unique issues identified issues in near-peer relationships.

Investigating feedback orientation in medical learners

BACKGROUND/PURPOSE

Feedback processes in health professions education (HPE) are not always successful. While recommendations to improve feedback provision dominate the literature, studying specific learner attributes that impact feedback uptake may also improve feedback processes. Feedback orientation is a concept from management science involving four dimensions of learner attributes and attitudes that impact their feedback uptake: utility, accountability, social awareness, and feedback self-efficacy. Feedback orientation may represent a valuable concept in HPE. We aimed to understand medical learners' feedback orientation at different stages in their development.

METHODS

We used the Feedback Orientation Scale, a 20-item survey instrument, for a cross-sectional analysis of feedback orientation in medical students and Internal Medicine residents at one large academic center. We performed descriptive statistics and analysis of variance for data analysis.

RESULTS

We found the same factors (dimensions) to feedback orientation in our population as in management science. Overall feedback orientation scores were high and were largely consistent across trainee levels. Utility was the domain that was highest across learners, whereas feedback self-efficacy was lowest.

CONCLUSIONS

Feedback orientation represents a useful concept to explore medical learners' attitudes toward feedback's role in their development. The four domains can help guide further nuanced feedback research and application.[Box: see text].

How progress evaluations are used in postgraduate education with longitudinal supervisor-trainee relationships: a mixed method study

The combination of measuring performance and giving feedback creates tension between formative and summative purposes of progress evaluations and can be challenging for supervisors. There are conflicting perspectives and evidence on the effects supervisor-trainee relationships have on assessing performance. The aim of this study was to learn how progress evaluations are used in postgraduate education with longitudinal supervisor-trainee relationships. Progress evaluations in a two-year community-pharmacy specialization program were studied with a mixed-method approach. An adapted version of the Canadian Medical Education Directives for Specialists (CanMEDS) framework was used. Validity of the performance evaluation scores of 342 trainees was analyzed using repeated measures ANOVA. Semi-structured interviews were held with fifteen supervisors to investigate their response processes, the utility of the progress evaluations, and the influence of supervisor-trainee relationships. Time and CanMEDS roles affected the three-monthly progress evaluation scores. Interviews revealed that supervisors varied in their response processes. They were more committed to stimulating development than to scoring actual performance. Progress evaluations were utilized to discuss and give feedback on trainee development and to add structure to the learning process. A positive supervisor-trainee relationship was seen as the foundation for feedback and supervisors preferred the roles of educator, mentor, and coach over the role of assessor. We found that progress evaluations are a good method for directing feedback in longitudinal supervisor-trainee relationships. The reliability of scoring performance was low. We recommend progress evaluations to be independent of formal assessments in order to minimize roles-conflicts of supervisors.

What makes giving feedback challenging? A survey of the Association Of Professors Of Dermatology (APD)

Feedback is an important aspect of resident education, so understanding the challenges that attendings face in giving feedback is important. To determine dermatology attendings' different perspectives on providing feedback to trainees and what barriers may exist in this process, an online survey study was emailed to members of the Association of Professors of Dermatology (APD) via its listserv; response was voluntary. Participants were dermatologists who are members of the Association of Professors of Dermatology. The main outcome measures were responses to survey questions that indicated comfort with feedback, perceived importance of feedback, and barriers to providing feedback. Of the 68 total survey respondents, 43(63%) were female and 25(37%) male. A majority of the attendings agreed that feedback has a positive influence on residents and improves patient outcomes. Many respondents reported giving regular feedback but a majority did not feel comfortable doing so with all residents in the program. Barriers to providing feedback included sensitivity on the part of the residents, lack of time, and fear of retaliation. When asked about ways to increase comfort, 12(17.6%) dermatology faculty noted that nothing would improve their ability to provide feedback while 28(41.2%) demonstrated interest in specialized training sessions. These results suggest that a majority of dermatology faculty face challenges in providing feedback to all of their trainees. Many would attend specialized trainings and have scheduled feedback sessions as part of the residency training to make both residents and attendings more comfortable with the whole process.

Self-Assessment in Feedback Conversations: A Complicated Balance

PURPOSE

Learner-centered feedback models encourage educators to ask learners to self-assess at the start of feedback conversations. This study examines how learners perceive and respond to self-assessment prompts during feedback conversations and assesses medical students' perceptions of and approach to self-assessment used as the basis for these conversations.

METHOD

All rising second-, third-, and fourth-year medical students at a midwestern U.S. medical school were invited to participate in this study. Students participated in 1-on-1 interviews between June and August 2019 during which they were asked open-ended questions about their experiences with self-assessment and feedback during medical school. The interviews were audio recorded and transcribed, and comments related to self-assessment in feedback conversations were extracted. Thematic analysis was used to identify recurrent ideas and patterns within the transcripts, and all excerpts were reviewed and coded to ensure that the identified themes adequately captured the range of student responses.

RESULTS

A total of 25 students participated in the study. Although some students noted improvement in their self-assessment abilities with increasing experience, no consistent gender, race, or training-level differences were found in reported attitudes or preferences. Students identified many benefits of self-assessment and generally appreciated being asked to self-assess before receiving feedback. Students had varied responses to specific self-assessment prompts, with no clear preferences for any particular self-assessment questions. Students described weighing multiple factors, such as image concerns and worries about impact on subsequent evaluations, when deciding how to respond to self-assessment prompts.

CONCLUSIONS

The process by which learners formulate and share self-assessments in feedback conversations is not straightforward. Although educators should continue to elicit self-assessments in feedback discussions, they should recognize the limitations of these self-assessments and strive to create a safe environment in which learners feel empowered to share their true impressions.

"Maybe it's the first time somebody's been honest with you": exploring how residents reconcile feedback variability

BACKGROUND

Supervisors in postgraduate medical education may deliver different feedback for the same quality of performance. Residents may struggle to make sense of inconsistent and sometimes contradictory information. We sought to explore how residents experience feedback from different supervisors, how they process inconsistent information, and what factors influence their experiences.

METHODS

Eighteen residents participated in semi-structured interviews to explore their perspectives on feedback. Using a constructivist grounded theory approach, we engaged in iterative cycles of data collection and analysis, sampling until theoretical sufficiency was reached. Constant comparative analysis was used to identify and define themes.

RESULTS

We identified a central theme of reconciliation, which we defined as the act of processing inconsistent feedback and determining how to engage with it. This reconciliation was informed by the credibility of, and residents' relationship with, supervisors and was achieved through conversations with peers and mentors, observation of other supervisors' behavior toward their performance, and reflection on their own performance. Participants expressed a reluctance to discard feedback, even if they felt it was incongruent with previous feedback or their own self-concept and self-assessment.

CONCLUSION

The findings of this study show that while residents are regular consumers of feedback, not all feedback is used equally. Residents actively reconcile sometimes-contradictory feedback and must work to balance a general reluctance to discard feedback, while developing an understanding of its credibility. This work reinforces the importance of pedagogical relationships and identifies that facilitated reflection that explicitly acknowledges feedback inconsistencies may be important in the reconciliation process.

Evaluating the Quality of Written Feedback Within Entrustable Professional Activities in an Internal Medicine Cohort

BACKGROUND

Whether written comments in entrustable professional activities (EPAs) translate into high-quality feedback remains uncertain.

OBJECTIVE

We aimed to evaluate the quality of EPA feedback completed by faculty and senior residents.

METHODS

Using retrospective descriptive analysis, we assessed the quality of feedback from all EPAs for 34 first-year internal medicine residents from July 2019 to May 2020 at Western University in London, Ontario, Canada. We assessed feedback quality on 4 domains: timeliness, task orientation, actionability, and polarity. Four independent reviewers were blinded to names of evaluators and learners and were randomized to assess each EPA for the 4 domains. Statistical analyses were completed using R 3.6.3. Chi-square or Fisher's exact test and Cochran-Armitage test for trend were used to compare the quality of feedback provided by faculty versus student assessors, and to compare the effect of timely versus not timely feedback on task orientation, actionability, and polarity.

RESULTS

A total of 2471 EPAs were initiated by junior residents. Eighty percent (n=1981) of these were completed, of which 61% (n=1213) were completed by senior residents. Interrater reliability was almost perfect for timeliness (κ=0.99), moderate for task orientation (κ=0.74), strong for actionability (κ=0.81), and moderate for polarity (κ=0.62). Of completed EPAs, 47% (n=926) were timely, 85% (n=1697) were task oriented, 83% (n=1649) consisted of reinforcing feedback, 4% (n=79) contained mixed feedback, and 12% (n=240) had neutral feedback. Thirty percent (n=595) were semi- or very actionable.

CONCLUSIONS

The written feedback in the EPAs was task oriented but was neither timely nor actionable. The majority of EPAs were completed by senior residents rather than faculty.

Can Rater Training Improve the Quality and Accuracy of Workplace-Based Assessment Narrative Comments and Entrustment Ratings? A Randomized Controlled Trial

PURPOSE

Prior research evaluating workplace-based assessment (WBA) rater training effectiveness has not measured improvement in narrative comment quality and accuracy, nor accuracy of prospective entrustment-supervision ratings. The purpose of this study was to determine whether rater training, using performance dimension and frame of reference training, could improve WBA narrative comment quality and accuracy. A secondary aim was to assess impact on entrustment rating accuracy.

METHOD

This single-blind, multi-institution, randomized controlled trial of a multifaceted, longitudinal rater training intervention consisted of in-person training followed by asynchronous online spaced learning. In 2018, investigators randomized 94 internal medicine and family medicine physicians involved with resident education. Participants assessed 10 scripted standardized resident-patient videos at baseline and follow-up. Differences in holistic assessment of narrative comment accuracy and specificity, accuracy of individual scenario observations, and entrustment rating accuracy were evaluated with t tests. Linear regression assessed impact of participant demographics and baseline performance.

RESULTS

Seventy-seven participants completed the study. At follow-up, the intervention group (n = 41), compared with the control group (n = 36), had higher scores for narrative holistic specificity (2.76 vs 2.31, P < .001, Cohen V = .25), accuracy (2.37 vs 2.06, P < .001, Cohen V = .20) and mean quantity of accurate (6.14 vs 4.33, P < .001), inaccurate (3.53 vs 2.41, P < .001), and overall observations (2.61 vs 1.92, P = .002, Cohen V = .47). In aggregate, the intervention group had more accurate entrustment ratings (58.1% vs 49.7%, P = .006, Phi = .30). Baseline performance was significantly associated with performance on final assessments.

CONCLUSIONS

Quality and specificity of narrative comments improved with rater training; the effect was mitigated by inappropriate stringency. Training improved accuracy of prospective entrustment-supervision ratings, but the effect was more limited. Participants with lower baseline rating skill may benefit most from training.

Guidelines: The Do's, Don'ts and Don't Knows of Creating Open Educational Resources

Background

In medical education, there is a growing global demand for Open Educational Resources (OERs). However, OER creators are challenged by a lack of uniform standards. In this guideline, the authors curated the literature on how to produce OERs for medical education with practical guidance on the Do's, Don'ts and Don't Knows for OER creation in order to improve the impact and quality of OERs in medical education.

Methods

We conducted a rapid literature review by searching OVID MEDLINE, EMBASE, and Cochrane Central database using keywords "open educational resources" and "OER". The search was supplemented by hand searching the identified articles' references. We organized included articles by theme and extracted relevant content. Lastly, we developed recommendations via an iterative process of peer review and discussion: evidence-based best practices were designated Do's and Don'ts while gaps were designated Don't Knows. We used a consensus process to quantify evidentiary strength.

Results

The authors performed full text analysis of 81 eligible studies. A total of 15 Do's, Don't, and Don't Knows guidelines were compiled and presented alongside relevant evidence about OERs.

Discussion

OERs can add value for medical educators and their learners, both as tools for expanding teaching opportunities and for promoting medical education scholarship. This summary should guide OER creators in producing high-quality resources and pursuing future research where best practices are lacking.

Residents' and supervisors' experiences when using a feedback-model in post-graduate medical education

BACKGROUND

Supervisors play a key part as role models and supporting the learning during residents' post-graduate medical education, but sometimes lack sufficient pedagogic training and are challenged by high demands in today's healthcare. The aim of this study was to describe the strengths and areas for improvement identified in the supervision process by residents and supervisors in post-graduate medical education.

METHODS

This study included supervisors and residents working at departments and health centres who have used a web-based questionnaire, as a part of the Evaluation and Feedback For Effective Clinical Teaching (EFFECT) model, during the period 2016-2019. Descriptive statistics and content analysis were used to analyse ratings and comments to describe strengths and areas for improvement in the supervision process.

RESULTS

The study included 287 resident evaluations of supervisors and 78 self-evaluations by supervisors. The supervisor as a role model, being available, and, giving personal support, were the three most important strengths identified by the residents and supervisors. Residents in primary care also identified the role modelling of general practice competence as a strength, whereas residents and supervisors in hospital departments addressed supervisors as energetic and showing work was fun. The area with the need of most improvement was, Giving and receiving feedback.

CONCLUSIONS

To be able to give feedback, residents and supervisors, needed to see each other in work, and the learning environment had to offer time and space to pedagogical processes, like feedback, to improve the learning environment.

Evaluating postgraduate family medicine supervisor feedback in registrars' learning portfolios

BACKGROUND

 Postgraduate supervision forms a vital component of decentralised family medicine training. While the components of effective supervisory feedback have been explored in high-income countries, how this construct is delivered in resource-constrained low- to middle-income countries has not been investigated adequately.

AIM

 This article evaluated supervisory feedback in family medicine registrars' learning portfolios (LPs) as captured in their learning plans and mini-Clinical Evaluation Exercise (mini-CEX) forms and whether the training district or the year of training affected the nature of the feedback.

SETTING

 Registrars' LPs from 2020 across five decentralised sites affiliated with the University of the Witwatersrand in South Africa were analysed.

METHODS

 Two modified tools were used to evaluate the quantity of the written feedback in 38 learning plans and 57 mini-CEX forms. Descriptive statistics, Fisher's exact and Wilcoxon rank-sum tests were used for analysis. Content analysis was used to derive counts of areas of feedback.

RESULTS

 Most learning plans (61.2%) did not refer to registrars' clinical knowledge or offer an improvement strategy (86.1%). The 'extent of supervisors' feedback' was rated as 'poor' (63.2%), with only 14.0% rated as 'good.' The 'some' and 'no' feedback categories in the mini-CEX competencies (p  0.001 to p = 0.014) and the 'extent of supervisors' feedback' (p  0.001) were significantly associated with training district. Feedback focused less on clinical reasoning and negotiation skills.

CONCLUSION

 Supervisors should provide specific and constructive narrative feedback and an action plan to improve registrars' future performance.Contribution: Supervisory feedback in postgraduate family medicine training needs overall improvement to develop skilled family physicians.

Feedback in family medicine clerkships: a qualitative interview study of stakeholders in community-based teaching

Feedback is an important aspect of teaching and learning in medical education. Irrespective of the training environment, too little effective dialogic feedback occurs. Community-based outpatient learning environments, such as general practitioner practices, have heterogeneous framework conditions regarding feedback that decrease feedback quality. To improve feedback in this setting, characteristics of feedback in such learning environments must be considered. This study aims to reveal such characteristics from different perspectives and derive ideas for improving feedback in community-based learning environments. Three stakeholder groups in family medicine clerkships as an example of community-based learning environments (n = 15 students, n = 12 faculty and administrative staff, n = 13 general physician trainers) were interviewed for this study. Transcripts of the interviews were analysed with qualitative content analysis. All stakeholders interviewed note a lack of feedback between groups. Feedback in primary care practices takes place in specific contexts (e.g., during vs after a consultation, during vs at the end of the clerkship) and is provided in different ways (e.g., verbal vs nonverbal). Barriers of effective feedback in community-based settings are: lack of opportunity/initiation, fear of giving feedback, unawareness (of correct feedback and/or lack of prior experience with feedback), and little basis for feedback. Currently, the exchange between the university and community-based learning environments is limited to grading and report writing, with little sharing of meaningful information. The potential of a better exchange between those within community-based learning environments and the university to improve feedback processes is not reached. This exchange and the framework conditions specific for the community-based learning environment should be considered as parts of the structural dimension of feedback. Teachers and course managers of family medicine institutes are in an important position to shape these factors actively, working together with stakeholders of community-based teaching.

Opportunities to Increase Workplace Feedback: A Multi-institutional Examination

OBJECTIVE

While feedback is an essential component of resident education, there are few large-scale studies examining when and under what conditions formative feedback is provided. Workplace-based assessment systems offer an opportunity to identify factors influencing when faculty provides feedback to trainees. Influential factors affecting feedback may provide targets for increasing and improving feedback in resident education.

DESIGN

Data on whether dictated feedback was provided were obtained from the Society for Improving Medical Professional Learning (SIMPL) mobile application. We used generalized linear mixed effects models to identify the degree to which faculty members, procedures, surgical case characteristics, and trainee performance were associated with whether narrative feedback was provided using SIMPL.

SETTING

This study was conducted using data from members of the SIMPL collaborative.

PARTICIPANTS

67,434 evaluations from 70 general surgery programs were included from 2015 to 2021. Of these, 25,355 evaluations included dictated feedback.

RESULTS

Approximately 61% of the variation in whether dictated feedback was provided was attributable to the individual faculty member. Compared to residents who achieved autonomy ratings of "Active Help," residents who achieved ratings of "Supervision Only" (odds ratio (OR) = 0.80, 95% confidence interval (CI) = 0.72, 0.88) had a lower likelihood of receiving dictated feedback. Residents who achieved ratings of "Intermediate" (OR = 0.81, CI = 0.74, 0.89), "Practice-Ready" (OR = 0.50, CI = 0.45, 0.57), or "Exceptional (OR = 0.64, CI = 0.54, 0.76) showed a lower likelihood of receiving dictated feedback compared to those rated as "Inexperienced." Cases rated as "High" in terms of complexity were associated with an increased likelihood of having dictation (OR = 1.35, CI = 1.26, 1.44).

CONCLUSIONS

The largest contributing factor for whether dictated feedback is included in a SIMPL evaluation are factors specific to the attending surgeon. Resident performance, resident autonomy, and case complexity had only modest associations with feedback decisions. Efforts to improve the amount of formative feedback for trainees should be directed towards reducing the variation in which attending surgeons elect to provide feedback.

Facilitating Residents' Understanding of Electronic Health Record Report Card Data Using Faculty Feedback and Coaching

PURPOSE

Feedback continues to present a challenge for competency-based medical education. Clear, consistent, and credible feedback is vital to supporting one's ongoing development, yet it can be difficult to gather clinical performance data about residents. This study sought to determine whether providing residents with electronic health record (EHR)-based report cards, as well as an opportunity to discuss these data with faculty trained using the R2C2 model, can help residents understand and interpret their clinical performance metrics.

METHOD

Using action research methodology, the author team collected EHR data from July 2017 to February 2020, for all residents (n = 21) in one 5-year Emergency Medicine program and created personalized report cards for each resident. During October 6-17, 2020, 8 out of 17 eligible residents agreed to have their feedback conversations recorded and participate in a subsequent interview with a nonphysician member of the research team. Data were analyzed using thematic analysis, and the authors used inductive analysis to identify themes in the data.

RESULTS

In analyzing both the feedback conversations as well as the individual interviews with faculty and residents, the authors identified 2 main themes: (1) Reactions and responses to receiving personalized EHR data and (2) The value of EHR data for assessment and feedback purposes. All participants believed that EHR data metrics are useful for prompting self-reflection, and many pointed to their utility in providing suggestions for actionable changes in their clinical practice. For faculty, having a tool through which underperforming residents can be shown "objective" data about their clinical performance helps underscore the need for improvement, particularly when residents are resistant.

CONCLUSIONS

The EHR is a valuable source of educational data, and this study demonstrates one of the many thoughtful ways it can be used for assessment and feedback purposes.

Gender differences in emergency medicine resident assessment: A scoping review

Background

Growing literature within postgraduate medical education demonstrates that female resident physicians experience gender bias throughout their training and future careers. This scoping review aims to describe the current body of literature on gender differences in emergency medicine (EM) resident assessment.

Methods

We conducted a scoping review which adhered to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews guidelines. We included research involving resident physicians or fellows in EM (population and context), which focused on the impact of gender on assessments (concept). We searched seven databases from the databases' inception to April 4, 2022. Two reviewers independently screened citations, completed full-text review, and abstracted data. A third reviewer resolved any discrepancies.

Results

A total of 667 unique citations were identified; 10 studies were included, and all were conducted within the United States. Four studies reported differences in EM resident assessments attributable to gender within workplace-based assessments (qualitative comments and quantitative scores) by both attending physicians and nonphysicians. Six studies investigating clinical competency committee scores, procedural scores, and simulation-based assessments did not report any significant differences attributable to gender.

Conclusions

This scoping review found that gender bias exists within EM resident assessment most notably at the level of narrative comments typically received via workplace-based assessments. As female EM residents receive higher rates of negative or critical comments and discordant feedback documented on assessment, these findings raise concern about added barriers female EM residents may face while progressing through residency and the impact on their clinical and professional development.

Teacher Guidance in the Digitalization Era: Efforts to Improve Student Achievement by Strengthening Online Sourced Learning Materials

The article aimed to analyze the teacher guidance in the digitalization era: an effort to improve student achievement by strengthening online-sourced learning materials at Madrasah Aliyah Negeri (MAN) Insan Cendikia Pelaihari Banjarmasin. Through semi-structured interviews with several teacher councils, the researchers have received many vital inputs regarding the dedication of teachers in guiding students to learn applications in the digital era in the distance learning situation.  The researchers used a phenomenological approach to understand how teachers monitor online learning activities in the current era of technology. The result showed that the teachers was able to get the students care and using learning resources in various forms of application. The dedication of the teachers and the students in Madrasah Aliyah teachers from Pelaihari Banjarmasin has become strengthening in online sourced learning materials. The stduents has changed to get learning achievement to be better result. Hopefully, these findings help developing educational studies in the current era of technology. Keywords: Online Learning Materials, Improve Students Achievement, Online Learning

Best Practices for Remediation in Pulmonary and Critical Care Medicine Fellowship Training

Background

Remediation of struggling learners in pulmonary and critical care fellowship programs is a challenge, even for experienced medical educators.

Objective

This evidence-based narrative review provides a framework program leaders may use to address fellows having difficulty achieving competency during fellowship training.

Methods

The relevant evidence for approaches on the basis of each learner’s needs is reviewed and interpreted in the context of fellowship training in pulmonary medicine and critical care. Issues addressed include bias in fellow assessments and remediation, the impacts of the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) pandemic, the specific challenges of pulmonary and critical care fellowship programs, a brief review of relevant legal issues, guidance on building and leveraging program resources, and a discussion of learner outcomes.

Results

This results in a concise, evidence-based toolkit for program leaders based around four pillars: early identification, fellow assessment, collaborative intervention, and reassessment. Important concepts also include the need for documentation, clear and written communication, and fellow-directed approaches to the creation of achievable goals.

Conclusion

Evidence-based remediation helps struggling learners in pulmonary and critical care fellowship to improve their ability to meet Accreditation Council for Graduate Medical Education (ACGME) milestones.

[Workplace-based Assessments: A Needs Analysis of Residents and Supervisors]

Workplace-based Assessments: A Needs Analysis of Residents and Supervisors Abstract. During residency training, four workplace-based assessments (WBA) are planned per year in the form of Mini-CEX and/or DOPS. They were introduced as a tool for giving feedback and defining learning objectives in the clinical setting. The aim of the present study is to identify facilitating and inhibiting factors. The results will provide information to improve the use of this learning tool to effectively promote learning in the workplace. First, all users must be trained in its use. In particular, it is important to provide immediate and specific feedback that identifies opportunities for improvement and sets achievable learning goals. Documentation should be user-friendly and provide an overview of the learning process. WBAs should not be perceived as a duty, but as a tool for valuable learning moments.