Faculty feedback that begins with resident self-assessment: motivation is the key to success

What would you like to ask your audience? Enhancing feedback value in PM&R grand rounds through customizable evaluation forms with presenter-generated questions

BACKGROUND

Feedback and evaluation are important in the professional development of academic physiatrists. Yet, physical medicine and rehabilitation (PM&R) learners giving academic presentations receive limited narrative feedback through generic evaluation forms.

OBJECTIVE

To assess whether customizable evaluation forms that integrate a presenter's specific questions would be associated with an increase in quantity and quality of narrative feedback received from the audience.

DESIGN

Separate samples pre-post intervention study.

SETTING

A large academic PM&R department's grand rounds.

PARTICIPANTS

PM&R faculty and trainees attending grand rounds (10-50 attendees with one presenter per session). The study included 20 presentations pre intervention (across 1 year) and 38 presentations post intervention (across about 3 years).

INTERVENTION

A customizable evaluation form that integrates a presenter's own questions into a tailored evaluation form comprising both standardized and presenter-built questions.

MAIN OUTCOME MEASURE(S)

Narrative feedback quantity was defined as the mean percentage and number of evaluation forms per presentation with at least one comment. Narrative feedback quality included three metrics: mean percentage and number of evaluation forms per presentation with comments that (1) contained ≥8 words, (2) referenced something specific, and (3) offered an actionable suggestion.

RESULTS

Compared to preintervention, presentations in the postintervention period had a greater mean percentage of evaluation forms containing at least one comment (pre = 33.4%, post =74.7%, p < .001), a comment that contained ≥8 words (pre = 20.2%, post = 44.2%, p < .001), a comment that referenced something specific (pre = 19.6%, post = 55.1%, p < .001), and a comment that offered an actionable suggestion (pre = 10.2%, post = 22.2%, p < .001).

CONCLUSIONS

Use of a customizable evaluation form in PM&R grand rounds that integrates a presenter's own questions was associated with a greater mean percentage of evaluation forms containing comments as well as comments meeting quality metrics related to length, specificity, and actionability.

Core Competencies Self-Assessment and Patient-Practitioner Orientation during the First Year of a Brazilian Orthopedic Residency

Objective  Training a competent physician requires to direct the resident profile of graduate students for practice activities. We sought to identify the doctor-patient relationship orientation and the self-assessment of the core competencies, which they pointed out needed to be developed. Methods  All 56 orthopedic residents admitted between 2016 and 2019 participated in the present prospective observational study. The Patient Practitioner Orientation Scale (PPOS) and a self-assessment questionnaire were answered at the beginning and end of the first year of residency (R1) in Orthopedics and Traumatology. We calculated mean and standard deviation for PPOS items and scores and analyzed them through the paired t-test. Self-Assessment Questionnaire answer options were "yes" or "I need to improve it" and skills were classified in decreasing order of the frequency of "I need to improve it" responses with description of absolute number and percentage. We compared frequencies using Fisher Test. P-values < 0.05 were considered statistically significant. GraphPad Prism 8.4.3 (GraphPad Software, San Diego, CA, USA) and Microsoft Excel (Microsoft Corporation, Redmond, WA, USA) were used for statistical analysis. Results  In the period between the beginning and the end of R1, the total PPOS mean score significantly decreased from 4.63 to 4.50 ( p  = 0.024), more biomedical-focused. Around one-third of the residents identified competencies of patient care, practice-based learning and improvement, and interpersonal and communication skills as needed to improve. Conclusions The PPOS and self-assessment activities could promote reflection practices and are possible tools for learner-centered competency assessment. Biomedical guidance tends to prevail as the training of physicians progresses, and periodic self-assessments can be worked on to build a growth mindset.

If we assess, will they learn? Students' perspectives on the complexities of assessment-for-learning

Introduction

Assessment can positively influence learning, however designing effective assessment-for-learning interventions has proved challenging. We implemented a mandatory assessment-for-learning system comprising a workplace-based assessment of non-medical expert competencies and a progress test in undergraduate medical education and evaluated its impact.

Methods

We conducted semi-structured interviews with year-3 and 4 medical students at McGill University to explore how the assessment system had influenced their learning in year 3. We conducted theory-informed thematic analysis of the data.

Results

Eleven students participated, revealing that the assessment influenced learning through several mechanisms. Some required little student engagement (i.e., feed-up, test-enhanced learning, looking things up after an exam). Others required substantial engagement (e.g., studying for tests, selecting raters for quality feedback, using feedback). Student engagement was moderated by the perceived credibility of the system and of the costs and benefits of engagement. Credibility was shaped by students' goals-in-context: becoming a good doctor, contributing to the healthcare team, succeeding in assessments.

Discussion

Our assessment system failed to engage students enough to leverage its full potential. We discuss the inherent flaws and external factors that hindered student engagement. Assessment designers should leverage easy-to-control mechanisms to support assessment-for-learning and anticipate significant collaborative work to modify learning cultures.

Evaluation of a workplace assessment method designed to improve self-assessment in operative dentistry: a quasi-experiment

BACKGROUND

Dental education has placed continued emphasis on self-regulated learning (SRL) and its subprocess, self-assessment. This study set out to evaluate the effectiveness of a novel workplace assessment method in developing trainees' self-assessment of operative procedures.

METHODS

A Direct Observation of Procedural Skills (DOPS) form was modified for the use and measurement of self-assessment. Participants were trained on how to conduct self-assessment using the designed assessment form and its grading rubric. Feedback and feedforward sessions were given to address self-assessment and performance issues. A P-value less than 0.10 was considered significant and the confidence level was set at 90%.

RESULTS

Thirty-two Year 5 dental students with an age mean of 22.45 (SD = 0.8) completed five self DOPS encounters during the clinical operative dentistry module in 2022. The aggregated total deviation (absolute difference) between self-assessment and teacher assessment decreased consistently in the five assessment encounters with a significant mean difference and a medium effect size (P = 0.064, partial Eta squared = 0.069). Participants' self-assessment accuracy differed from one skill to another and their ability to identify areas of improvement as perceived by teachers improved significantly (P = 0.011, partial Eta squared = 0.099). Participants' attitudes towards the assessment method were positive.

CONCLUSIONS

The findings suggest that the self DOPS method was effective in developing participants' ability to self-assess. Future research should explore the effectiveness of this assessment method in a wider range of clinical procedures.

Self-Assessment in Feedback Conversations: A Complicated Balance

PURPOSE

Learner-centered feedback models encourage educators to ask learners to self-assess at the start of feedback conversations. This study examines how learners perceive and respond to self-assessment prompts during feedback conversations and assesses medical students' perceptions of and approach to self-assessment used as the basis for these conversations.

METHOD

All rising second-, third-, and fourth-year medical students at a midwestern U.S. medical school were invited to participate in this study. Students participated in 1-on-1 interviews between June and August 2019 during which they were asked open-ended questions about their experiences with self-assessment and feedback during medical school. The interviews were audio recorded and transcribed, and comments related to self-assessment in feedback conversations were extracted. Thematic analysis was used to identify recurrent ideas and patterns within the transcripts, and all excerpts were reviewed and coded to ensure that the identified themes adequately captured the range of student responses.

RESULTS

A total of 25 students participated in the study. Although some students noted improvement in their self-assessment abilities with increasing experience, no consistent gender, race, or training-level differences were found in reported attitudes or preferences. Students identified many benefits of self-assessment and generally appreciated being asked to self-assess before receiving feedback. Students had varied responses to specific self-assessment prompts, with no clear preferences for any particular self-assessment questions. Students described weighing multiple factors, such as image concerns and worries about impact on subsequent evaluations, when deciding how to respond to self-assessment prompts.

CONCLUSIONS

The process by which learners formulate and share self-assessments in feedback conversations is not straightforward. Although educators should continue to elicit self-assessments in feedback discussions, they should recognize the limitations of these self-assessments and strive to create a safe environment in which learners feel empowered to share their true impressions.

"Maybe it's the first time somebody's been honest with you": exploring how residents reconcile feedback variability

BACKGROUND

Supervisors in postgraduate medical education may deliver different feedback for the same quality of performance. Residents may struggle to make sense of inconsistent and sometimes contradictory information. We sought to explore how residents experience feedback from different supervisors, how they process inconsistent information, and what factors influence their experiences.

METHODS

Eighteen residents participated in semi-structured interviews to explore their perspectives on feedback. Using a constructivist grounded theory approach, we engaged in iterative cycles of data collection and analysis, sampling until theoretical sufficiency was reached. Constant comparative analysis was used to identify and define themes.

RESULTS

We identified a central theme of reconciliation, which we defined as the act of processing inconsistent feedback and determining how to engage with it. This reconciliation was informed by the credibility of, and residents' relationship with, supervisors and was achieved through conversations with peers and mentors, observation of other supervisors' behavior toward their performance, and reflection on their own performance. Participants expressed a reluctance to discard feedback, even if they felt it was incongruent with previous feedback or their own self-concept and self-assessment.

CONCLUSION

The findings of this study show that while residents are regular consumers of feedback, not all feedback is used equally. Residents actively reconcile sometimes-contradictory feedback and must work to balance a general reluctance to discard feedback, while developing an understanding of its credibility. This work reinforces the importance of pedagogical relationships and identifies that facilitated reflection that explicitly acknowledges feedback inconsistencies may be important in the reconciliation process.

Partnering With Residents on the Redesign of the Internal Medicine Resident Self-Evaluation Form

INTRODUCTION

The positive impact of resident-driven synthesis of assessment data has been associated with increased intrinsic motivation to learn and create an individualized strategy to improve performance. The objective of the study was to incorporate residents' recommendations for restructuring the self-assessment metric into a tool that will promote a well-organized and effective self-improvement plan.

MATERIALS AND METHODS

Residents and faculty collaborated on pre- and post-intervention questionnaires to assess the barriers to the timely completion of the current self-evaluation form and gather information on the tool's ability to stimulate the formation of concrete goals. The residents were also invited to provide their recommendations on the structure of the new tool and the educational domains that were assessed by the tool. The post-survey also evaluated the capacity of the proposed tool to guide residents in establishing specific goals.  Results: The new form is concise and more precise in assisting the learner in developing short-term and long-term goals and the strategies and resources to achieve them.  Discussion: Collaborating with the learners created an opportunity to address the faculty's and residents' most important concerns about the effectiveness of the metric.

CONCLUSION

 In a learner-centered model, resident participation is critical in designing/redesigning a practical self-assessment tool for residents in Internal Medicine.

Part 5: Essentials of Neonatal-Perinatal Medicine Fellowship: evaluation of competence and proficiency using Milestones

The Accreditation Council for Graduate Medical Education (ACGME) Pediatric Subspecialty Milestone Project competencies are used for Neonatal-Perinatal Medicine (NPM) fellows. Milestones are longitudinal markers that range from novice to expert (levels 1-5). There is no standard approach to the required biannual evaluation of fellows by fellowship programs, resulting in significant variability among programs regarding procedural experience and exposure to pathology during clinical training. In this paper, we discuss the opportunities that Milestones provide, potential strategies to address challenges, and future directions.

Integrating self-assessment into feedback for emergency medicine residents

BACKGROUND

In 2013 the Accreditation Council for Graduate Medical Education (ACGME) introduced "Milestones" designed to nationally standardize the assessment of resident physicians. Previous studies compare resident self-assessment on milestones to faculty assessment, with varying degrees of agreement, but integration of self-assessment into the formative feedback process has not yet been directly studied. This study uses a conceptual framework of self-determination theory, integrated with concepts from adult learning theory, to compare the perception of the feedback quality given in semiannual reviews before and after the incorporation of resident self-assessment into the feedback process.

METHODS

This was an interventional study conducted in a single emergency medicine residency program at a major academic hospital over 1 calendar year. Residents first engaged in a semiannual review without self-assessment. At subsequent semiannual reviews, residents completed a Milestone-based self-assessment that was provided to the faculty member assigned to conduct their semiannual review. Residents and faculty completed surveys rating perception of feedback quality. Two-sided Wilcoxon signed-rank tests were used in comparison analysis.

RESULTS

One resident did not self-assess prior to the semiannual review and was excluded leaving 25 paired surveys for analysis. Residents found feedback after the self-assessment more actionable (p = 0.013), insightful (p = 0.010), and better overall (p = 0.025). Similarly, faculty felt the feedback they provided was more actionable (p < 0.001), more insightful (p < 0.001), and better communicated (p < 0.001); led to improved resident understanding of milestones (p < 0.001); and were overall more satisfied (p < 0.001). Free-text comments explore pre- and postintervention perceptions of feedback.

CONCLUSIONS

Integration of self-assessment into semiannual reviews improves perception of feedback given to residents as perceived by both residents and faculty. Although limited by sample size, the results are promising for a simple, evidence-based intervention to improve feedback during an existing mandated feedback opportunity.

"When in Doubt, Ask the Patient": A Quantitative, Patient-Oriented Approach to Formative Assessment of CanMEDS Roles

INTRODUCTION

Since the introduction of competency-based frameworks into postgraduate medical curricula, educators have struggled to implement robust assessment tools that document the progression of necessary skills. The global movement towards competency-based medical education demands validated assessment tools. Our objective was to provide validity evidence for the Ottawa CanMEDS Competency Assessment Tool (OCCAT), designed to assess clinical performance in the communicator, professional, and health advocate CanMEDS roles.

METHODS

We developed the OCCAT, a 29-item questionnaire informed by specialty-specific Entrustable Professional Activities and consultation with stakeholders, including patients. Our sample included nine neonatal-perinatal medicine and maternal fetal medicine fellows rotating through antenatal high-risk clinics at the Ottawa Hospital. Following 70 unique encounters, the OCCAT was completed by patients and learners. Generalizability theory was used to determine overall reliability of scores. Differences in self and patient ratings were assessed using analyses of variance.

RESULTS

Generalizability analysis demonstrated that both questionnaires produced reliable scores (G-coefficient > 0.9). Self-scores were significantly lower than patient scores across all competencies, F(1, 6) = 13.9, p = .007. Variability analysis demonstrated that trainee scores varied across all competencies, suggesting both groups were able to recognize competencies as distinct and discriminate favorable behaviors belonging to each.

DISCUSSION

Our findings lend support to the movement to integrate self-assessment and patient feedback in formal evaluations for the purpose of enriched learner experiences and improved patient outcomes. We anticipate that the OCCAT will facilitate bridging to competency-based medical education.

Feedback Credibility in Healthcare Education: a Systematic Review and Synthesis

PURPOSE

The purpose of this study was to systematically review and synthesize factors that influence learners' perceptions of credibility when feedback is provided by an authority figure in a healthcare environment.

METHODS

This study reviewed literature from medicine, psychology, and education using systematic review and qualitative synthesis methods. In a multi-step process, major electronic bibliographic databases were searched for relevant studies until October 2020.

RESULTS

The search identified 9216 articles. A total of 134 abstracts underwent full-text review. Of these, 22 articles met inclusion criteria. The studies were heterogenous and the majority utilized a qualitative design with interviews and focus groups. A few studies employed mixed methodology (n = 2) and two studies used a quantitative design. Four main themes were identified: feedback characteristics, context of feedback, source credibility, and recipient characteristics.

CONCLUSION

As programs implement major educational change initiatives to create more formative assessment practices, feedback will become even more crucial. The four main themes identified are important factors that contribute to the perception of feedback credibility. While the factors are described independently, they may be viewed as interrelated and the association between these factors and feedback may be driven more by learning culture than each characteristic.

SUPPLEMENTARY INFORMATION

The online version contains supplementary material available at 10.1007/s40670-020-01167-w.

Effects of Feedback on Midwifery Students' Self-Assessed Performance and Their Self-Assessment Ability: A Quasi-Experimental Study

This study aimed to evaluate the effects of verbal and written feedback in clinical midwifery placement on students' self-assessed performance and their self-assessment ability. This three-group quasi-experimental study was conducted on 120 students. Participants in the control group received clinical education through the routine method, while in the feedback groups received either verbal or written feedback methods on the basis of the sandwich feedback model. In the last day of clinical education, a checklist was simultaneously filled out by participants and a second instructor. There was significant direct correlation between the scores of performance assessment by both the second instructor and students in the control group (r = .38, p = .01), the verbal feedback group (r = .63, p < .001), and the written feedback group (r = .74, p < .001). The rates of student-instructor agreement in the control, verbal feedback, and written feedback groups were 32.5%, 70%, and 77.5%, respectively. Feedback is effective in significantly improving students' self-assessment ability.

Twelve tips for clinicians dealing with uncertainty when assessing learners

Clinician educators often experience distress caused by uncertainty regarding how effectively to participate in assessment practices in a way that supports both their programs and their students. Uncertainty is a common state for clinicians, particularly for those who see patients with early or ill-defined illness presentations. While clinicians often feel ill at ease when facing uncertainty in the clinical realm, becoming comfortable with uncertainty and learning to manage such states are now recognized as vital components of clinical practice. Clinicians, as a result, have adopted a series of strategies to lessen the unease that uncertainty can create. While similar experiences plague clinician educators placed in assessment roles, much less attention has been given to how we can support individuals in the education setting. Here, the distress of uncertainty may be greater due to clinician educators having less experience with assessment practices. Fortunately, strategies that are effective in the clinical domain can be translated into the assessment realm to accommodate uncertainty when assessing learners. In this 12 tips article we offer guidance on the translation of such strategies.

Constructing a Shared Mental Model for Feedback Conversations: Faculty Workshop Using Video Vignettes Developed by Residents

INTRODUCTION

Providing feedback is a fundamental principle in medical education; however, as educators, our community lacks the necessary skills to give meaningful, impactful feedback to those under our supervision. By improving our feedback-giving skills, we provide concrete ways for trainees to optimize their performance, ultimately leading to better patient care.

METHODS

In this faculty development workshop, faculty groups used six feedback video vignettes scripted, enacted, and produced by residents to arrive at a shared mental model of feedback. During workshop development, we used qualitative analysis for faculty narratives combined with the findings from a focused literature review to define dimensions of feedback.

RESULTS

Twenty-three faculty (physical medicine and rehabilitation and neurology) participated in seven small-group workshops. Analysis of group discussion notes yielded 343 codes that were collapsed into 25 coding categories. After incorporating the results of a focused literature review, we identified 48 items grouped into 10 dimensions of feedback. Online session evaluation indicated that faculty members liked the workshop's format and thought they were better at providing feedback to residents as a result of the workshop.

DISCUSSION

Small faculty groups were able to develop a shared mental model of dimensions of feedback that was also grounded in medical education literature. The theme of specificity of feedback was prominent and echoed recent medical education research findings. Defining performance expectations for feedback providers in the form of a practical and psychometrically sound rubric can enhance reliable scoring of feedback performance assessments and should be the next step in our work.